Appendix V — Deficiencies

Intervention components found to be deficient by one or more authors during qualitative interviews, as described in chapter 10.

V.1 Include design, features, and language to foster trust

Relevant website features: Professional design. EQUATOR’s Logo remains prominent. Citation metrics are presented at the top the reporting guidance. Information about who developed the guidelines, how they developed it, and why the guidance is credible is still provided, and easily findable from the top of the guidance.

Barriers addressed: Researchers may not believe stated benefits

See text

V.2 Describe what reporting guidelines are where they are first encountered

Relevant website features: Prominent definition on home page and guideline page.

Barriers addressed: Researchers may not know what reporting guidelines are

See text

V.3 Use tone of voice and design to communicate personal benefits; confidence and simplicity

Relevant website features: A clean, simple interface for the home page and guidance pages. Text uses phrases like “confidence”, “quick”, “maximum impact”.

Barriers addressed: Researchers may not believe stated benefits

See text

V.4 Describe personal benefits and benefits to others where reporting guidelines are introduced (home page, on resources, in communications)

Relevant website features: Benefits are prominently and consistently displayed across the home page and guidance pages. Descriptions prioritize personal benefits to the authors above hypothetical benefits to others.

Barriers addressed: Researchers may not know what benefits to expect

See text

V.5 Clarify what tasks (e.g., writing, designing, or appraising research) guidelines and resources are designed for

Relevant website features: Clear instruction and differentiation of resources

Barriers addressed: Researchers may not know what reporting guidelines are; Researchers may not know when reporting guidelines should be used

See text

V.6 For each item, provide examples of reporting in different contexts

Relevant website features: SRQR already had some examples. No more examples added

Barriers addressed: Researchers may not know how to report an item in practice

See text

V.7 Define key terms

Relevant website features: SRQR now has a glossary, and text is marked-up with definitions that appear upon click.

Barriers addressed: Researchers may misunderstand

Participants wanted more definitions within the guidance and on the rest of the website

Some complex words in the redesigned SRQR guideline had blue dotted lines underneath. When participants clicked them, a definition would pop up. Participants stressed usefulness of this feature, and everybody liked that the definitions appeared in a popup, and not on a new page.

“I got great help there for me as a writer.” (ECR from Ecuador)

“One feature that I really found to be very useful was where you [described] different kind of kinds of terms there […] When I’m not quite sure about the term there was. If I clicked on it, there was a description and that was very clearly written out and it made it quite easy to use the information that was there.” (Pre-PhD student from Malawi)

Participants wanted more definitions within the guidance (some struggled with words like “transferability” or “generalizability”) and other areas of the website. For example, when reading about the scope of SRQR and related guidelines, participants struggled to understand terms like “qualitative evidence synthesis”

“A qualitative evidence synthesis. What’s that? Is that like a review?” (Researcher from South Africa)

When reading a list of guidelines for different study types on the home page, one participant explained how not understanding terms like “cohort study” led her to feel anxious and worried.

“Actually, I feel a little anxious because I’m not sure whether it shows that I am not good at something or I am ignorant or something. So when I saw all of these and, like half of them, I didn’t know. I may feel anxious and maybe, like, lose confidence sometimes.” (PhD student from China)

User experience might be clearer if definitions were signified by a different colour (not blue)

Because I wanted to know whether future users would discover this functionality by themselves, I asked participants what they expected the dotted lines to signify. Most participants guessed correctly.

“Interviewer: What do you expect those dotted lines to mean? Participant: Umm, I think dotted lines would mean that it will give me a definition.” (Researcher from Australia)

“It’s not exactly a hyperlink, but it might give you a definition or something like that” (Researcher from the UK)

However, a couple of participants thought the lines might signify hyperlinks, and another thought it was a misspelt word because it looked like Microsoft Word’s autocorrect feature.

“I’m honestly not sure whether I see just a highlight or it’s a link. Usually links are in blue, completely in blue.” (Medical student from Ghana)

“I think uh, there are some misspelled words” (ECR from India)

V.8 Cater to different kinds of user (readers vs dippers) by structuring guidance with headings, itemisation, hyperlinking to particular sections, and with optional content

Relevant website features: SRQR items are structured consistently, making information easier to find. Itemisation is used consistently, content is hyperlinked when useful.

Barriers addressed: Researchers may expect the costs to outweigh benefits

Overall, participants welcomed the structure and navigation features, including headings, consistent subheadings for each guideline item, navigation menus, and links to sections.

“I find the guideline more interactive than simply, you know, a PDF document. So I really appreciate that it was more structured and the guide was more easy to follow. And somehow having all these other, umm options, you know that the web offers, yeah […] For me it was very useful for moving around more easily and more quickly” (ECR from Ecuador)

Participants liked having extra content as notes, but footnotes might not be optimal implementation

When an SRQR item contained extra context or explanation not relevant to all users, I moved this information to a footnote, but participants expected the footnote identifiers (superscript numbers) to be references, and did not like how reading the footnote took them to the bottom of the page, away from where they had been reading:

“I think it’s a citation. So uh linked to a reference. But I think it may also be foot note.” (Researcher from the UK)

“[Superscript numbers are] always a link to, like, a reference. If not a reference like some…[clicks it]…okay, so it’s telling me things. It’s probably not a reference like in Wikipedia. It shows you it’s a link. This one is telling me more things. So it’s a footnote.” (Research consultant from the Philippines)

“I worked hard to scroll down. I’m not gonna go back up. So, whatever, i’m not reading [it].” (Research consultant from the Philippines)

Menu navigation was useful but could be more prominent and go one level deeper

The redesigned SRQR guideline has a side navigation menu so users can easily navigate to the Introduction, Methods, Results, and Discussion guidance sections, or to the FAQs and citation information. A few participants did not notice the menu, or suggested it could be more prominent, and one suggested it would be useful if the menu listed all reporting items (i.e.,. not just Methods, but also the item titles within).

“when I scrolling I forgot to pay attention to the menu on the side. So […] I suddenly realized it was a menu there […] Maybe make it much more, like, the characters larger or, like, the colour, maybe change it, because now it is so light and I didn’t pay attention to that. But this menu is useful if I pay attention to this.” (PhD student from China)

Each reporting item has its own menu within the collapsible content, so users can jump to the item’s justification, examples, or related resources. The same participant described these item menus as a useful way to help “find the detailed part” they desired.

Section URLs could be more intuitive

When the mouse hovers above a reporting item heading, a button would appear to allow users to a copy the URL to that section. One participant did not find this intuitive, and thought the button would take them to more information instead.

Participants may desire a summary of the guidance

In the think aloud, one participant said they would prefer to browse the checklist above the full guidance because “in my head, like the checklist probably has these things already” (Research consultant from the Philippines). They expected the checklist to act as a summary. This expectation may explain why another participant said “I just want to see what this thing is and at the moment I don’t know. Should I read the guidelines or checklist?”, perhaps because they desired an overview (Researcher from South Africa).

“So maybe not the full guideline, but I’m not sure how how you can present it. Maybe one paragraph, or, uh, or you could present a mind map. Maybe you can present a mind map and you can put the keywords of this guideline. So if people want to see, for example, […] introduction, literature review, methodology, blah blah. Something like this.” (PhD student from China)

Participants found item titles hard to spot when scrolling

“The names of the sections are, like, black and a little plain. So I think […] the name of each section should be somehow highlighted” (ECR from Ecuador)

“And they don’t really stand out in my opinion, like, just about the layout of the page themself. You see, this [heading] doesn’t stand out so much from the rest, just the font is a little bigger. For example, if I’m scrolling through it very fast, it may be I might miss it. So if those headings or subheadings could be made more visible…” (Medical student from Ghana)

Some participants did not expect / want all guidance on one page

“I didn’t expect it to be part of the same page because, just looking at the side [menu] here, it makes it a very long page to scroll through. So I guess what I’m a bit more familiar with is, you know, you click on something, it takes you to the guidance and then each of [the items] might be on a separate page […] So it’s less of a scroll and more of a click.” (Researcher from the UK)

V.9 Explain how the guidance was developed and why it can be trusted

Relevant website features: Brief description included on home page and at top of reporting guideline, links to full to development information

Barriers addressed: Researchers may feel patronized

The relationship between the website and publication could be clearer

The top of the guideline page references the original SRQR publication. Some participants said this helped them trust the website, but others voiced confusion about the relationship between the publication and webpage, whether the guidance matched, and which they should cite.

“I know I have the information to look for the paper and I could find it. So yeah, it inspires trust and it’s amazing to have it here.” (ECR from Ecuador)

“Is this [content on the website] now this article? Because if it is the article then I’m not clear on that […] [After reading through the item titles] Do they match?…Yes, they match.” (Researcher from South Africa)

The FAQs helped clear up this relationship, and some participants recommended they be signposted (or briefly explained) at the top of the guidance page.

“[The FAQ section about development] gave me more information about how [the guideline] came about. You know, I was asking [for] that when we first spoke, and here it was. It was nice to be able to see [that] somebody put this together. […] I like that I could get more information about the background. So in a way it’s not really a frequently asked question. It is actually the background.” (Researcher from South Africa)

“I think it’s very fine to put it here [in the FAQs], but I’m just thinking about, like, could we put a link or a summary at the very beginning of the website? Because it may make me trust it much more quickly.” (PhD student from China)

V.10 Make guidance appear shorter by removing superfluous information, hiding optional content, splitting long guidelines, using concise language, and separating design advice

Relevant website features: SRQR has been edited. The only text presented immediately is instruction on what the author needs to describe. Additional information is hidden at first and can be expanded. Text is shortened through editing and by using active voice. In the case of SRQR, this reduced the text length by 60%.

Barriers addressed: Researchers may expect the costs to outweigh benefits

On first impression, the guidance may appear too long

Participants spoke positively of the various features to hide optional content (like collapsible sections and pop-ups). Nobody expressed wanting all information up front, and all preferred to have it only when needed.

“If you want to, you can extend and read through [the extra information]. I think that’s very useful.” (Medical student from Ghana)

“but if you don’t need [the extra information], uh, you can’t see it. But if you need it, you may click on it and just show it very directly.” (PhD student from China)

However, some participants still felt the guidance appeared very long, and this could be off-putting upon first impression.

“So I kind of know how long this page is based on seeing [the scroll bar] move. I don’t want to read the whole thing […] I don’t want to spend 17 min on that” (Research consultant from the Philippines)

In the first iteration, the page was even longer because of content in the introduction before the guidance began (e.g., information about the guideline’s scope, development, or how to use it).

“There’s so much going on [at the top of the page, before the guidance]. […] Yo, this is getting annoying. All I want is the guideline or the document” (Researcher from South Africa)

In the second iteration, I moved some of this introduction information into a collapsible box, but some participants still described the page as very long.

V.11 Create spaces for authors to discuss reporting guidelines with others

Relevant website features: Each reporting item has its own discussion board.

Barriers addressed: Researchers may not believe stated benefits; Researchers may misunderstand; Researchers may not know how to report an item in practice; Reporting guidelines can become outdated; Researchers may feel patronized

Each reporting item had a link to its own discussion page. Participants welcomed the opportunity to post comments, ask questions, and suggest edits on this page.

“So you do not want to just show people how to do it, but you want to provide a platform for the readers to discuss and to create their own ideas. I think that’s great.” (PhD student from China)

“Maybe what helps is if you can, say, share your thoughts here, […] you feel a little bit more [that] I’ve got a voice instead of I’m just being told that it’s a great place to be and it’s got all the answers.” (Researcher from South Africa)

“Maybe if I identify something that I know is not useful for me. Yeah, in that case, maybe I would write something, you know, to other researchers to read it and maybe give feedback about it, but also for developers of the guideline.” (ECR from Ecuador)

The button could be clearer

The link to the discussion page was a button with an icon of a speech bubble, no text, and not all participants knew what this button would do or where it might take them. One participant said this button would be easier to understand if it used text, and recommended it be placed at the end of the expandable content for each item.

“Maybe the blog? No, I don’t know.” (ECR from Ecuador)

“I thought it might be, uh, like a frequently asked questions or a or a further explanation” (Researcher from Australia)

“I think you can just put [a button] after all the [extra] content you have seen [in the expandable box], because if if people haven’t seen this content they will have nothing [to discuss].” (PhD student from China)

The discussion page could be easier to use

The discussion page requires users to sign in before they can post. This deterred some participants.

“Ohh I see I need to sign in. Ah, usually that is something that discourages me immediately.” (Medical student from Ghana)

“I would comment, if I didn’t have to log in” (Research consultant from the Philippines)

Additionally, one participant did not understand the discussion page’s purpose.

“Am I doing this to myself? Is it comments about my research that I’m completing the guidance for or is it for the website for other people to see these comments that I’ve made perhaps?” (Researcher from the UK)

V.12 Address communications to authors

Relevant website features: All resources and website copy are directed predominantly at authors.

Barriers addressed: Researchers may feel that checking reporting is someone else’s job.

Participants expected the target audience was researchers, but the target could be more specific

Everyone recognised the website was for researchers, as opposed to editors. Most realised it was for health researchers.

“It can be useful for academics. It can be useful for people who do research on daily basis or as a regular part of their job” (ECR from India)

Some participants specified the target might be early career researchers or university students.

“those who are just getting started in the career journey in research….[pause]…definitely students?…umm…. people who don’t really have that much experience in writing.” (Pre-PhD student from Malawi)

“Maybe like master research or higher degrees and also for some junior scholars” (PhD student from China)

“But I think your your target audience is completely different than the very basic student…because it’s more for the expert researchers, right… for the expert researchers” (ECR from India)

Some participants hinted that seeing terms specific to their field signalled that the website was for them:

“If I know any of these terms, then clearly I’m gonna make use of it” (ECR from India)

Ohh then maybe if it sounded like “Want help writing up Qualitative research or something like that, so that I know that this is actually addressing me. It’s relevant to me. Yeah, but [the way it is written now] way is kind of open. So it’s not kind of, it’s not relating to me. (Pre-PhD student from Malawi)

V.13 Structure guideline items to make them quicker to digest

Relevant website features: Items have consistent structure and use bullet points consistently

Barriers addressed: Researchers have limited time

Participants welcomed items’ structure and said they preferred it to unstructured text.

“It was more structured and and the guide was more easy to follow.” (ECR from Ecuador)

One participant questioned whether they might prefer full sentences over bullet points. They used SRQR’s item 12 as an example, where the text says:

“If the actual sample differs from the target sample, describe:

  • the difference,
  • why these differences may have occurred,
  • how this might affect the findings.”

The participant suggested changing this to “Why the difference has occurred and how it might affect the findings. So one line and you’re done with it”, instead of taking up four lines of text.” (ECR from India)

V.14 Describe the scope of a reporting guideline at the top of every resource

Relevant website features: The intended scope of a guideline is clearly & prominently described. This definition includes contexts in which the guidance should not be used.

Barriers addressed: Researchers may not know whether a reporting guideline applies to them

Not all participants found the scope clear

One participant was not clear whether SRQR applied to survey studies, and said they wanted a definition of what qualitative research meant. Another participant wondered why the guidance discussed patient outcomes when the scope had not specified health research.

V.15 Search function on website

Relevant website features: Search function is easier to find as a recognizable icon in the navigation bar of every page. The home page includes additional ways to access search functionality.

Barriers addressed: Guidance may be difficult to find

The website featured search buttons (one in the navigation menu, one at the top of the home page), but the search buttons did not work, so participants could not explore the search functionality. However, many participants commented on the search buttons and instinctively knew what they were.

V.16 Include testimonials from research users who benefit from complete reporting

Relevant website features: SRQR includes dummy testimonials and quotes from research users

Barriers addressed: Researchers may not care about the benefits of using a reporting guideline

Quotes helped participants believe items were important, but some participants questioned their credibility

The redesigned SRQR guidance featured quotes in the margin of some items from people that use research, like evidence synthesisers, editors, or other researchers. These quotes were made up, as I didn’t have time to collect real ones just for testing. Some participants said they liked these quotes:

“I think it makes this….how can I say?…practical from a different point of view. Like why exactly you need this [reporting item]. So now this person [in the quote] is telling from her own perspective how useful it is that you have [the item] described clearly, so it makes it such that if I’m trying to describe [this item], I’ll try to keep that in mind that.” (Medical student from Ghana)

“I like it because each of them tells me the why…umm…you know, kind of gives a plain language reason for […] why it’s a useful thing. […] That’s gives it, you know, humanity.” (Researcher from Australia)

However, some participants questioned whether these quotes were from real people.

“Maybe they’re real…maybe it’s legit” (Research consultant from the Philippines)

Quotes may not be valuable enough to be prominent

Some participants didn’t find the quotes useful, or described them as distracting.

“I don’t care what people think.” (Researcher from South Africa)

“I didn’t really pay much attention to it. I saw a few quotes and read through a few quotes, but I was like, OK, so what does that add to me?” (Pre-PhD student from Malawi)

“sometimes when I have to read [something complicated], but I don’t want to, [distraction] may be a big problem for me because, like, these comments are what people say, so it’s, like, more easy to read than the [reporting item]. So it may attract a lot of attention from me” (PhD student from China)

V.17 Include testimonials from researchers who were nervous about being punished for reporting transparently

Relevant website features: Quotes included alongside guideline

Barriers addressed: Researchers may expect the costs to outweigh benefits

Some of the fictitious quotes contained reassurance from researchers who had felt nervous when using a reporting guidelines because they felt unsure about being so transparent about parts of their work they knew were not perfect. There were only a couple of quotes of this kind, and few authors noticed or commented on them.

“That makes it relatable to a user, particularly a new user, because we can see that all of these people are, you know, they were first time users once.” (Researcher from Australia)

V.18 Instruct authors to cite reporting guidelines so readers may learn about them

Relevant website features: Consistent instruction to cite reporting guidelines

Barriers addressed: Researchers may not know what reporting guidelines exist

The title How to cite was misleading to one participant

Participants expected citation instructions and most were not surprised to see them in a section called How to cite. However, one participant questioned whether the section would tell them how to cite the guidance, or how to cite resources in general.

“I want to see, like, if it says right away, a certain citation style like, are you talking about Chicago?” (Research consultant from the Philippines)

Some participants did not know whether to cite the website or the publication

One participant said they had sought the original paper because they “wanted to add the citation” (ECR from Ecuador) to their article. Another asked “Do you reference the EQUATOR network or do you reference the original article?” (Researcher from South Africa).

Some participants recommended the citation instruction be more prominent

“I think you can put citation in a in a box in a [coloured] box to make people notice it.” (PhD student from China)

“It is helpful of course, umm, but I think it got lost in the page.” (ECR from Ecuador)

V.19 Explain when reporting guidelines do not intended to prescribe structure

Relevant website features: Explained at top of guidance

Barriers addressed: Researchers may struggle to reconcile multiple sets of guidance

Some participants included sub headings for each reporting item in their writing sample

Many participants noticed and welcomed the clarification that SRQR does not prescribe structure. Nobody objected to it.

“I think this is very clear. If you say that did not prescribe order nor structure. […] Yes, that’s great. […] I like it very much because I think it helps me understand that the guideline is not a standardized one, but it depends on the user. To use this kind guide in their own context. So I think this one this sentence is is very important.” (PhD student from China)

In the writing task, two participants used subheadings for the items they wrote. When asked about these subheadings in the second interview, both said they intended to remove at least some of the subheadings. One participant replied “One of our reviewers wanted more structure, so of course we leave some [subheadings] in, but not all” (ECR from Ecuador). Another said “I’ll probably not include them [in my final article], but the information, that’s what I’m I’m going to maintain, yeah.” (Pre-PhD student from Malawi)

V.22 Educate authors about writing as a process

Barriers addressed: Researchers may not consider writing as reporting

The website did not sufficiently address participants’ need for writing training

The top of the SRQR guideline page had short instructions about how to apply the reporting guideline whilst writing but for many participants, this was not enough. Many participants expressed desire for training on how to write an article (as opposed to what to write).

One participant wanted training on how to “write in a concise manner” (Pre-PhD student from Malawi)

Another talked about their struggle to apply training and guidance to their own writing process.

“When I try to look at your guidance on your website, I really want to use it in my own writing, but it is very strange because when I try to, uh, connect the information on the website with my own writing, I found there there might be a great gap because I think everything on your website is very clear (actually they are very specific, those suggestions), but when I try to connect those information with my own writing, I found it just a little bit difficult to generate some some specific ideas to start my writing.

So I’m thinking if that’s because the problem of my writing is not the lack of specific guidance but some other thing like my motivation or, I don’t know…it’s just a little bit strange.

And I also talk about this with my friends because lots of my friends, they are also PhD students and they are struggling at writing too. So ask them if they have some guidance, uh, if they have looked at some guidance and if [they] have put those guidelines in [their] own writing and and their answers were, like, quite similar with me and and they all talk about that, ‘yes, we we look at lots of guidance we try to look at lots of those writing books to teach you how to write, to teach you how to structure your writing. But it’s still very hard’. When you really sit down and start writing, actually you couldn’t, uh, call up [the information].” (PhD student from China)

V.23 For each item, explain why the information is important and to whom (not just what constitutes “good” design)

Relevant website features: Information added when necessary

Barriers addressed: Researchers may not know why items are important

The labels Justification and Why readers need this information were ambiguous to some participants

Each reporting item included a section within its expandable content called Why readers need this information alongside examples of writing and links to other resources. Participants found the information useful but were less enthusiastic than they were for the examples.

“So yeah, this this section, justification, examples and resources. I found them very useful as they give more detailed explanations on each specific section, and particularly the examples” (Medical student from Ghana)

However, a few readers interpreted the title of this subsection differently.

“I thought it might give you an example of how to justify [the choices you made in your study]” (Researcher from the UK)

“I was asking myself: which readers?…. the one like me who is using the platform or the readers of my paper?” (Midwifery student from Uganda)

Not all participants found the justification compelling

Whereas many participants spontaneously said how useful examples were, none said the same about the justification section.

“[It was] one of those things that you just read and go, like, ohh OK. But not, like, something which I requested or you sit down and think through.” (Pre-PhD student from Malawi)

Interviewer: Maybe this section could do a better job of reminding authors who is going to be reading their research, and how different those people might be. Or their different perspectives or experiences. Do you think if it had done that you would have found it a bit more convincing and motivating?

Participant: Umm. Definitely, definitely. From your explanation, I think it kind of puts it in context, [as to] why this section is more important. So I didn’t read it at first, but I think it did not come up very clearly as compared to how you’ve explained it. And so, after hearing that explanation, I’d say that this the way the readers need information.” (Pre-PhD student from Malawi)

V.24 For each item, provide clear instruction of what needs to be described

Relevant website features: Writing instruction occurs first for each item.

Barriers addressed: Researchers may not know how to report an item in practice

One participant felt the instructions could be a little longer and less ambiguous

Each reporting item began with instructions of what to write. Participants welcomed these instructions, as articulated by one participant when describing how they used the guideline for the writing task, “I used it specially for understanding what kind of information, you know, apart from the obvious ones, every section wants me to write, so it it was really helpful in that respect” (ECR from Ecuador).

However, one participant suggested the information could be a little longer: “I guess just a few more words to kind of, articulate [the instruction] better or give it a bit more detail [because it may be ambiguous to] somebody less experienced” (Researcher from the UK).

V.25 Gather and communicate evidence for benefits

Relevant website features: Dummy quotes provides evidence for experienced benefits.

Barriers addressed: Researchers may not believe stated benefits

Participants questioned whether quotes were credible

The website included (fake) quotes from authors exalting the personal benefit reporting guidelines have brought to their job (e.g., easier writing, smoother publishing). Some participants found these compelling, but others worried these quotes were biassed, fake, or vague.

“This is encouraging because it’s a feedback from a person working at the journal with responsibility for publishing. The editor.” (Midwifery student from Uganda)

“If you’re using quotes on a website, my first thought is you could be biasing, you know, how useful the website is, because you’ve picked out great quotes so, you know, until I’ve looked at this and used it myself, I might not agree” (Researcher from South Africa)

“Maybe you can put some concrete stories, because for me I like to read stories instead of just reading some uh, broad words like I like it. Yeah.” - (PhD student from China)

“Yes, I think it’s much more useful [to] have some timelines to show when people say [things], yeah, because if I could see the time, I may also check like how long the company or the website may [have] last[ed] for […], how many people use that and how they are feeling like from one year ago or two years ago to now.” (PhD student from China)

V.26 Provide advice regarding how to respond if asked to remove reporting guideline content by a colleague, editor, or reviewer

Relevant website features: Advice given in FAQ

Barriers addressed: Researchers may be asked to remove reporting guideline content

Many participants did not notice the advice

The FAQ section included some advice on what to do if a colleague or editor asked you to remove content from your manuscript pertaining to one or more SRQR items. Only one participant noticed this advice. When talking about what they would do if asked to remove content by “a reviewer or, even a co-author, it was amazing for me that you give these tips, you know, to what to do in those cases and what you can do to, umm, make it or to highlight the importance of have each section within your paper.” (ECR from Ecuador)

Another participant, who had not noticed the advice, nevertheless said they would return to the website should they be asked to remove content:

“I think first I will ask his or her reason why he or she want me to delete that if I think, I don’t agree with that. I may try to make some formal explanation to explain that and also I may come back to this website to double check some reliable things which can support my view to let them believe that I should also I I need this paragraph.” (PhD student from China)

V.27 Use plain language

Relevant website features: SRQR is edited to use plainer language.

Barriers addressed: Researchers may misunderstand

Participants did not understand all words

I had tried to use plain language on the home page and introduction to SRQR, but participants still questioned the meaning of some words (e.g., “synthesis”), including words commonly used by EQUATOR staff (e.g., “transparency”). Even the term “reporting guidelines” confused some participants, especially when used at the start of a sentence, where some participants interpreted the word “reporting” to be a verb, instead of part of a compound noun.

V.29 Reassure that all research has limitations to encourage explanation over perfect design

Relevant website features: This reassurance appears on the home page and all guidance pages

Barriers addressed: Researchers may expect the costs to outweigh benefits; Researchers may feel afraid to report transparently

Two participants spoke about content that reassured authors to be honest about limitations. After noticing a reassuring quote from an editor, one participant agreed, saying:

“sometimes when we don’t report limitations and the reviewer identifies the limitations [instead] then then we lose credibility. So it’s better we report [limitations] so that the the reviewers say ohh this person has acknowledged this limitation, then then this is a good study. So I just liked it.” (Midwifery student from Uganda)

Another voiced support for the SRQR reporting item about limitations, saying “no study is ever done perfectly or done in a way that the next person would do it. And so we’ve become really, really good, I think, in my team, at making quite long limitation sections to try and avoid peer reviewers from finding everything that’s wrong”. (Researcher from South Africa)

V.30 Tell authors when to use reporting guidelines, or that reporting guidelines are best used as early as possible

Relevant website features: Stated prominently

Barriers addressed: Researchers may not know when reporting guidelines should be used

Upon realising reporting guidelines are for writing articles, all participants naturally thought about using them in the drafting/writing process (as opposed to retrospectively check manuscripts that are already written). Sometimes this was immediate:

“From what I have seen [after the 5 second test], I think probably the website should be about, uh, helping you try to discover… identify the guidelines that you will use for writing your study quickly.” (Midwifery student from Uganda)

For some participants, using the guidance for the writing task reaffirmed their opinion that the guidelines would help them in early stages of writing.

“I saw it as very important for me at this stage that I’m at. I’m writing a manuscript. But I’d say that it can be usable at any level, only that I found it very important at my level, which is at my paper writing stage.” (Pre-PhD student from Malawi)

No participants talked about consulting reporting guidelines when planning a study

Perhaps because participants did not intuitively understand how they could use a reporting guideline to plan research, none talked about using them in that way.

V.31 Use consistent terms

Relevant website features: The website uses terms consistently

Barriers addressed: Researchers may misunderstand

Participants were confused when different words referred to the same thing

Two participants questioned why the home page referred to “guidelines” and “reporting guidelines” and asked whether there was a difference. Another asked whether “guidance” and “guideline” were the same. Another participant noticed that an SRQR item 18 uses the word ‘integration’ in two different contexts: whereas the item asks authors to integrate their work with others’, the example uses the word ‘integrating’ differently when discussing how their study combined modes of teaching.

V.32 Avoid patronizing language

Relevant website features: Continue to avoid using patronizing language

Barriers addressed: Researchers may feel patronized

No participants mentioned feeling patronized. When asked, one participant described the tone as “suitable” and said “simple comments” and “explaining like this, [is not patronizing] because I also need a lot to learn.” (PhD student from China)

V.33 Centralised hosting

Relevant website features: A core set of frequently accessed guidelines are now presented on a single website.

Barriers addressed: Guidance may be difficult to find

That the website contains many different reporting guidelines was rarely mentioned, but one participant said “to have a repository where all those things are, instead of having to go and search for them. That makes sense.” (Researcher from South Africa)

V.34 Provide translations

Relevant website features: Translations are prominently listed above the guidance

Barriers addressed: Researchers may not understand the language; Researchers may misunderstand

One participant noticed that their language was missing

“So what about, you know, all the researchers that speak Spanish?” (ECR from Ecuador)

V.35 Tell authors how long the guidance will take to read

Relevant website features: Estimated reading time given

Barriers addressed: Researchers have limited time

Not all participants liked being told how long the guidance would take to read

The guidance advises readers that it may take 16 minutes to read. One participant liked this: “Oh, I know how it’s going to take. It’s helpful when you are performing research you have like tight deadlines and not much time.” (ECR from Ecuador)

But another felt like 16 minutes was too long, especially if you only found the guideline as part of manuscript submission “Uh, it will be frustrating. Now again, you waste a lot of time.” (Midwifery student from Uganda).

And a third (not a native English speaker) worried they would feel bad if the guidance took them longer than the stated time:

“I think this is, like, useful, but I’m not sure whether some… when some people read it, they may feel stressful because, from my teaching experience before, some of my students may say to me if, like, there is guidance saying that you may need, like, 60 minute to read it, sometimes, if they take longer, they may feel confused, or lose a little bit [of] confidence [and worry that] they read it so slow or they are not, like, normal one. I think this [estimated time] may be a good [information] because, like, nowadays I found, like, many websites use it. But actually, for me sometimes this part is useful, but sometimes it is not.” (PhD student from China)

V.36 Encourage explanation even when choices are unusual or not optimal

Relevant website features: Added to items

Barriers addressed: Researchers may feel afraid to report transparently

Only one participant noticed this instruction

Some reporting items in SRQR ask authors to explain their reasoning behind design choices. Only one participant noticed one of these sentences. They reflected that researchers often “follow this path of only mentioning [what we did] but not explaining how we did it and why it was important to apply this strategy” (ECR from Ecuador).

V.37 Explicitly state when no better guidance exists for a use case

Relevant website features: Reporting guidelines warn authors when no better guidance exists for a use case, and how the current guidance can be adapted instead

Barriers addressed: Researchers may not know what reporting guideline is their best fit

Only one participant commented on this instruction

In the introduction to SRQR, where its scope is explained, the instruction mentions that there are no better reporting guidelines for writing protocols for qualitative research, and instead recommends authors use certain items from SRQR. One participant annotated this explanation and said they liked it (PhD student from China).

V.38 Provide instruction as to how and where information can be reported without breaching word count limits or making articles bloated.

Relevant website features: Added instruction at top of reporting guideline and in some items where most useful

Barriers addressed: Researchers may struggle to keep writing concise

Only one participant commented on instructions of where content can be reported

Very little of the guideline text deals with where content can be reported (e.g., in the article body, in an appendix, a table, a figure). Only one participant noticed it and said “I like the reminder” (Researcher from South Africa).

V.39 Reassure when guidelines are just guidelines

Relevant website features: Prominently displayed at top of reporting guideline

Barriers addressed: Researchers may feel restricted if reporting guidelines prescribe design

Only one participant drew the distinction that reporting guidelines are not rules

Throughout the homepage and SRQR page I had tried to convey that reporting guidelines are recommendations, and I took care not to use words like rules or standards. Only one participant commented explicitly about this, but no participants talked about the guideline as if it were a set of rules.

“So I think this explanation here is very clear that it helps researchers to know that definitely they can have their own ideas and this guideline is, it is kind of like a supporting one, but not a rule, not a standardized rule.” (PhD student from China)

V.40 Ensure all resources and tools (e.g., checklists and templates) are in ready-to-use formats

Relevant website features: No changes made

Barriers addressed: Researchers have limited time; Reporting guideline resources may not be in usable formats

Even though the links to the checklist and template were not live, participants expected the resources to be in ready-to-use formats.

V.41 Decrease fear of judgement by making reporting guidelines design agnostic

Relevant website features: SRQR explicitly states that it makes no assumptions about design. Inadvertent design assumptions were edited.

Barriers addressed: Researchers may expect the costs to outweigh benefits; Researchers may feel afraid to report transparently; Researchers may feel restricted if reporting guidelines prescribe design

No participants commented on this.

V.42 Explain importance of complete reporting to the scientific community

Relevant website features: Continue to do this

Barriers addressed: Researchers may not know why items are important

The website explains the societal and community importance of complete reporting in a few places: the justification subsections of each item, quotes in the margin, and briefly on the home page.

Although participants commented on the quotes from research consumers, and on the Justification sections within each reporting item, nobody talked about the negative impact of poor reporting on the scientific community at scale.

V.43 Provide clear instruction of what needs to be described when an item was not done, could not be done, or does not apply

Relevant website features: Instructed where relevant

Barriers addressed: Researchers may not know what to write when they cannot report an item

This component was not tested and no participants commented on this.

V.44 Remove branding and messaging that may invoke feelings of judgement, complexity, or administrative red-tape

Relevant website features: A clean, simple interface for the home page and guidance pages. Text makes less use of judgemental phrases and fewer references to the negative consequences of poor reporting.

Barriers addressed: Researchers may expect the costs to outweigh benefits

No participants described the design as unpleasant or judgemental.

V.45 Search Engine Optimization

Relevant website features: The site has additional meta-data. Each reporting guideline page has its own meta-data. The site is optimized for mobiles.

Barriers addressed: Guidance may be difficult to find; Researchers may not encounter reporting guidelines early enough to act on them

This component was not tested and no participants commented on this.

V.46 Use if-then rules to direct authors to more appropriate and up-to-date guidance when available

Barriers addressed: Researchers may not know what reporting guideline is their best fit

Although participants commented on the links to related guidelines, they did not comment on the “if…then…” structure of these links.