Contents

Find and evaluate Data Sources, Published Research & Literature Review Strategies

Contents

Find and evaluate Data Sources, Published Research & Literature Review Strategies

Nowadays, there are various ways to find and organize literature and data online. While a simple Google Scholar search may seem sufficient to quickly get an idea about a specific topic, this strategy becomes increasingly less feasible when trying to collect and evaluate evidence in an unbiased manner.

In this lecture, we will discuss:

  • How to find data sources/literature; How to find published research based on a specific data source

    • using Google Scholar, Web Of Science, PubMed

  • Basic search strategies

    • Forward/Backward search

  • Literature reviews

    1. Elaboration of a Review Question

    2. Elaboration of a Review Protocol

    3. Searching for All Eligible Studies

    4. Unbiased Screening of Eligible Studies

  • Literature maps

  • Digital tools for literature search/review

    • ResearchRabbit & more

  • Meta-analysis tools

    • Neurosynth/Neuroquery

  • Evaluate, aka. read a scientific paper

  • Finding data

How to find data sources/literature; How to find published research based on a specific data source

Where to find scientific literature

We’ll start off by briefly discussing the three most popular literature search engines; follow the links to get a more in-depth understanding of each engine.

Google Scholar:

Google Scholar has easily become the go-to search engine for scientific literature and is a good place to start to find a specific paper easily. Unfortunately, it offers less advanced search options than the two options below.

It is a free search engine providing access to scholarly literature, including articles, books, conference proceedings, and theses. It is a popular tool for researchers, students, and academics, as it offers a simple and user-friendly interface for discovering and accessing a comprehensive library of scholarly literature across most, if not all, disciplines. It also offers features such as citation tracking and alerting, which allows users to monitor and track the impact of their own research, as well as the work of others in their field. While the impact is not something we should focus on too much, it helps to identify the seminal papers on a given topic.

To work with Google Scholar, simply access the platform through the Google Scholar website or through their Google account. They can use the search bar to enter keywords and other search terms and then refine their results using the various filters available. Users can also set up alerts to be notified when new articles or publications related to their areas of interest become available.

google_scholar.png

Google Scholar generally provides links to full-text articles, when available, as well as other related resources, such as author profiles and citation metrics. It adds some additional functionality for the specific search results:

  • Clicking on the “All versions” keyword below a search result link will help you identify other versions of the paper if, e.g., the first source is paywalled.

  • Via the “Cite” keyword, you can also easily get a quick citation in a reference style of your choice.


Clicking on the 3 Bars in the top right, next to the search bar, reveals the Scholar menu. Here, you can select advanced search, which will open a pop-up looking like this:

scholar_advanced.png

It’s pretty self-explanatory, but it allows you to match phrases either in the title or text, exclude phrases, and filter by author, publication, and year range.

The advanced menu also allows you to set up E-mail notifications should new literature related to your field of interest be published. To set notifications, expand the menu on the top left, click on Alerts, click the Create Alert button, and enter your search criteria, as well as an e-mail address.

To remove an alert, simply move to the alters menu as above and click the Cancel link behind the alert.

created_alert.png

Additionally, if you start publishing, it’s a good idea to set up a profile to track your own publications and citations, making it a useful tool for managing your scholarly online presence and output.

Web of Knowledge/Sciene

The Web of Knowledge/Sciene is a powerful online research database that provides access to a wide range of scholarly literature, including articles, conference proceedings, and patents. Developed by Clarivate Analytics, it offers a suite of tools for researchers, academics, and professionals to discover, analyze, and manage research data.

One of the key benefits of Web of Knowledge is its comprehensive coverage of a vast array of disciplines, including science, technology, social sciences, and humanities. It also provides users with advanced search options, including citation searching and author searching, allowing them to identify influential works and authors in their fields of interest.

One of the problems of the platform is the need for a personal account and access through your institutional library. But if you’re willing to deal with this, you are rewarded with very rich (but somewhat confusing) advanced search options. The platform also offers training and support resources to help users deal with this and get the most out of their research.

Youtube: Web of Science training

Intoduction course Web of Science

Web of Science Platform Training & Support

PubMed

PubMed is a free online database of biomedical literature maintained by the United States National Library of Medicine. It offers access to millions of articles from thousands of journals, as well as books, conference proceedings, and other scientific resources in the fields of medicine, healthcare, and life sciences.

One of the key benefits of PubMed is its vast coverage of biomedical literature, making it a valuable resource for researchers, healthcare professionals, and students. It offers a range of search options, including keyword searches, author searches, and advanced search options that allow users to narrow down their results by date, article type, and other criteria.

PubMed also provides links to full-text articles, when available, as well as other related resources, such as clinical trials and systematic reviews. It also offers tools for citation analysis and tracking the impact of research articles.

To work with PubMed, you can access the platform through the National Library of Medicine website. Simply use the search bar to enter keywords and other search terms, and then refine your results using the various filters available. PubMed also offers a variety of tutorials, videos, and other resources to help users get the most out of their research.

Pubmed User Guide

Andvanced search tutorial

Local Libraries (shocked Pikachu face)

Most libraries offer access to their databases, as well as e-books and journals that the university subscribes to. Simply google your university library, e.g., the online presence of the library of the Goethe-University Frankfurt.

You’ll have to rely on the information provided by the library in question to find out how to work with their database. E.g.

Goethe-University Frankfurt: Tips for literature search

You can also write or visit your local library/librarian. You’d be surprised how well-versed librarians are in finding and managing information! (Well, it’s literally their job, but you know what I mean).


Basic search strategies

There are two basic strategies one can follow when searching for relevant literature: the forward search using keywords/phrases ' or the backward search following the references from a specific paper.Backward search can also involve reverse footnote mining/citation search,` i.e., searching for other sources that have cited a particular article.


General tips on optimizing your search behavior

It’s recommended to explore multiple combinations of search terms and operators during the literature search and document every search query so that you or others can reproduce your literature search later on. If you’re not taking notes for a literature review, this can still be relevant info that you should include in your lab notebook.

Further, you should use multiple search engines/databases for your literature search and tailor your search queries and vocabulary for each individual database.

Literature reviews

Systematic reviews & meta-analyses are used to evaluate the published literature on a given topic to gain a solid understanding of scientific consensus and degrees of evidence for certain positions. While this is not necessary for every paper you’ll be writing, it is a good idea to get comfortable with the basic ideas behind systematic reviews and how they correct search behavior for potential biases. This will also help you organize your literature search and make sure that you do not miss any important papers or waste time on papers that are irrelevant to your specific question/topic.

We’ll focus on the paper of Pigot & Polanin (2020): Methodological Guidance Paper: High-Quality Meta-Analysis in a Systematic Review. and for a more clinical/neuroscience related perspective the work of Bolzan & Oliveira (2021): A compact guide to the systematic review and meta-analysis of the literature in neuroscience.

And explore the following steps:

  1. Elaboration of a Review Question

  1. Elaboration of a Review Protocol

  1. Searching for All Eligible Studies

  1. Unbiased Screening of Eligible Studies


Bolzan & Oliveira (2021): A compact guide to the systematic review and meta-analysis of the literature in neuroscience.


The first step when searching for literature should be defining your research question, i.e., defining what exact topic or method we’re trying to explore.


1: Elaboration of a Review Question

We’re mostly interested in research questions, but these need to be necessarily defined into review questions once we start systematically searching for literature on a specific topic.

This can be done with the help of the mnemonic tools to turn a complex question into simple, comprehensive, and direct terms. In clinical and neuroscience, the following mnemonic tools could be used to formulate a specific review question.

PICO (“Has the intervention, I, changed the outcome, O, in the population, P, compared to control treatment, C?”)

SPICE (S – Setting; P –Population; I – Intervention; C – Comparison; E – Evaluation)

SPIDER (S – Sample or population of interest; PI – Phenomenon of Interest; D – Design; E – Evaluation; R – Research type)


Following this example, you could also devise your own mnemonic tool suited to the specific needs of your research question.

2: Elaboration of a Review Protocol

After defining our review question, we’ll construct a review protocol.

A review protocol serves as a comprehensive and structured plan that outlines the methodology and procedures that will be followed during a systematic review. The purpose of a review protocol is to provide a transparent and rigorous framework that ensures the systematic review is conducted in a consistent, reliable, and reproducible manner.

The protocol typically includes a clear research question, explicit inclusion and exclusion criteria for the selection of studies, the search strategy and databases that will be used, data extraction methods, quality assessment criteria, data synthesis and analysis methods, and a plan for reporting/documenting the results of your literature review.

At the heart of the review protocol are the search criteria, as well as the screening procedures. These will be explained in detail in the next section.

Pigot & Polanin (2020): Methodological Guidance Paper: High-Quality Meta-Analysis in a Systematic Review.

**Systematic reviews follow three basic steps: **

1. searching the literature

2. screening abstracts and full-text documents

3. Coding included studies

Each of these three basic steps should be documented as much as possible and has to demonstrate that the researcher has included all eligible studies on a specific topic/question. We’ll focus mostly on the first two points in the following chapter.

We’ll explore the basic strategies of systemic review to help ensure that we are thorough and unbiased in the review and selection of potential papers of interest.

3: Searching for All Eligible Studies

A search must necessarily be systematic, comprehensive, and well documented to allow for the reproducibility of the search process and to make sure that no relevant results were excluded.


  • systematic - method behind the search:

When conducting a search for relevant studies, it’s important to include all of the following:

  • Terms: relevant keywords to identify the research topic.

  • Strings/Phrases: combining terms using logical operators to refine the search

  • Limiters: applying specific filters to limit the search results to the most relevant

  • Tools: utilizing tools such as Boolean operators, proximity searching, and wildcards to fine-tune the search strategy.

  • Databases: selecting the most relevant databases to search for the study.

Terms and phrases need to be sensitive enough to capture all relevant studies, meaning that, e.g., the terms used to capture as many of the studies relevant to the topic as possible. In a literature search, it’s recommended to tailor your search terms/syntax to maximize sensitivity (the proportion of studies identified divided by the total number of relevant studies in existence) while being prepared to accept low precision (the proportion of relevant studies to the total number of studies identified) (Higgins et al., 2022). This is necessary as an increase in sensitivity will naturally lead to a reduction in precision,

The search strategy should be informed by the review question and can be based on the mnemonic tool used.

E.g., for the PICO tool, the search terms can be documented based on the number of retrievals (n) for all relevant terms (Bolzan & Oliveira, 2021):

  1. P: Terms related to (population P or synonyms) = (n(p))

  2. I: Terms related to (intervention I or synonyms) = (n(i))

  3. C: Terms related to (control C or synonyms) = (n(c))

  4. O: Terms related to (outcome O or synonyms) = (n(o))

  5. PICO: Combination of terms used in individual searches, i.e., (population P or synonyms) and (intervention I or synonyms) and (control C or synonyms) and (outcome O or synonyms) = (n(pico))

comprehensive - breadth of the search:

Require terms unique to several disciplines. include both online databases that index published literature as well as sources such as Google Scholar and Web of Science also includes strategies such as retrospective reference harvesting, prospective forward citation searching, and contacting prominent or active authors in the field a further attempt to identify unpublished literature, such as dissertations and reports from independent research firms

Further, we have to attempt to identify unpublished literature, such as dissertations and theses or reports from independent research firms. This is necessary to identify and ideally reduce publication bias, known as the phenomenon of studies published in research journals to report larger effect sizes and proportionally more statistically significant effects as should be expected (Polanin, Tanner-Smith, & Hennessy, 2016)

There exist in-depth protocols and tools that aim to estimate the effect of publication bias on a specific topic:


Cochrane: A revised tool to assess risk of bias in randomized trials (RoB 2)

Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach

To evaluate the publication bias, you may also use the Replicability Index, which provides a number of tools and information on how to spot problematic trends in published research.

well documented :

The documentation of your review process should Include all

  1. The review question

  2. Brief introduction to the research subject

  3. Description of the strategies employed to obtain publications and filter relevant studies

  4. All search queries, including which site was used and the date of the search

You can document your search process using different methods; the easiest is to include your search in a Lab-Notebbook.

4: Unbiased Screening of Eligible Studies

After identifying all relevant studies on a given topic, it’s necessary to screen the abstracts, full text, and citations of the studies in question.

The screening process involves the creation of screening tools and applying them to different aspects of the selected literature.

First is the creation of a screening tool/strategy to filter study abstracts and titles for ineligible articles such as essays or non-empirical studies. This helps you organize and sort the abstracts based on how likely it is that they should be included in your review.

The screening tool consists of a checklist of multiple clear and concise questions that lead to the inclusion or exclusion of a study. Following Polaning et al. (2019), The questions/ items should be

* (a) objective, 
* (b) “single-barreled, i.e., only related to a single aspect of the abstract: E.g., “Does the abstract indicate that participants were sampled from the general population?”
* (c) use the same sentence structure,
* (d) include yes/no/unsure answers only

This process can include free text-mining software like Abstrackr (Wallace et al., 2012).

After completing the title and abstract screening, gather all the included full-text PDFs. This step is commonly called retrieval.

The full-text screening process is similar to the abstract screening process and involves the following steps: * Developing a screening tool * Screening each article * Making a decision about whether it should be included

The questions in your screening tool should be informed by your research question and can include topics such as the methods used or the sampled population. Unfortunately, there is no validated, dependable text-mining tool to assist with the full-text screening process at the moment.

The resulting collection of studies should then be a solid base for future research. An easy way to keep track of the literature in question is to add them to a citation manager such as Zotero. A so-created Zotero library can be used to create a list of references on the fly, shared online, or integrated with a number of other digital tools. Find out more about how to work with Zotero in the chapter Project design.

Further Reading

The following studies provide example protocols of screening strategies in neuroscience:

Ramos-Hryb AB, Bahor Z, McCann S, et al. Protocol for a systematic review and meta-analysis of data from preclinical studies employing forced swimming test: an update. BMJ Open Science. 2019;3:e000035 https://openscience.bmj.com/content/3/1/e000043.abstract

Bolzan JA, Lino de Oliveira C. Protocol for systematic review and meta-analysis of the evidence linking hippocampal neurogenesis to the effects of antidepressants on mood and behaviour. BMJ Open Science. 2021;5:e100077 https://doi.org/10.1136/bmjos-2020-100077

Hohls JK, Konig H, Quirke E, Hajek A. Association between anxiety, depression and quality of life - a systematic review of evidence from longitudinal studies. PROSPERO 2018 CRD42018108008 Available from: https://www.crd.york.ac.uk/prospero/display_record.php ?ID=CRD42018108008

Excercise:

Create a Review question
Create a Review Protocol
Write a table for the documentation of your search process containing:
    1. The review question
    2. Brief introduction to the research subject
    3. Description of the strategies employes to obtain publications and filter relevant studies
    4. All search queries, including which site was used and the date of the search

Literature maps

A newer, graph-based approach to literature search is the creation and use of literature maps. If there is one takeaway from this lesson, it should be the use of literature maps for your future research!

https://media1.giphy.com/media/icgArcntfH5C0/giphy.gif?cid=ecf05e477osuhygwt3o5mqowvvtu798736t9f5gh0g6nlddq&rid=giphy.gif&ct=g

A literature map is a visual representation of the relationships between different topics, themes, and concepts in a specific field of study. It can be used to explore the connections and gaps between different areas of research, identify key authors and publications, and gain a broader understanding of the research landscape.

You could do this by hand, but this would be rather time-consuming, so we’ll introduce some tools for the creation of literature maps in the next section.

png depicting the file structure of the course template repository

Literature maps are useful because they help researchers and students to:


>

Identify gaps in the research: By visualizing the relationships between different topics and subfields, literature maps can help to identify areas where there is a lack of research or where new research could make a significant contribution.

Discover new connections and relationships: Literature maps can help to uncover new connections and relationships between different topics and fields, leading to new insights and avenues for research.

Gain a broader perspective: By providing a visual overview of the research landscape, literature maps can help researchers and students gain a broader perspective on their field of study and see how their own work fits into the larger context.

To make the most of a literature map, it is important to:


Choose the right tools: There are a variety of tools available for creating literature maps, ranging from simple mind-mapping software to more complex data visualization tools. Choose a tool that is appropriate for your needs and skill level.

Start with a clear research question: A literature map is only useful if it is focused on a specific research question or topic. Start by defining your research question or area of interest, and then use the literature map to explore the relationships between different concepts and subfields.

Be critical: While literature maps can be a useful tool for exploring the research landscape, it is important to be critical of the data sources and assumptions that underlie the map. Always evaluate the quality and relevance of the sources you are using, and be aware of any biases or limitations in the data.

Collaborate: Literature maps can be a valuable tool for collaboration, allowing researchers and students to share their knowledge and insights and work together to explore new connections and relationships between different topics and fields.

Digital Tools for Literature Search/Review

While it’s always sensible to think about how to develop a review question and follow a pre-defined search strategy “by hand,” Nowadays, there are quite a few tools out there to help you make your search more efficient/organized.

Most of the following tools automatically include literature maps and the ability to search different databases. While they have slightly different functionalities and foci, they are rather similar, so it’s best to just test what works best for you.

Researchrabbit:

To get started, we’ll explore one of these tools more closely.

The free, self-proclaimed “most powerful discovery app ever built for researchers!” and “new Spotify for academic papers” is an AI-based app that allows you to search for literature in multiple databases. It mainly works on the idea of structuring your literature into collections for every separate project.

ResearchRabbit builds visual representations of your collections, showing the relative importance of paper, as well as the connections between different papers based on, e.g., citations (where arrows always point towards the paper cited).


png depicting the file structure of the course template repository

Exploring the literature map

It improves the process of manually adding papers to your collection by automatically suggesting related works and authors. That is, as soon as a paper is added to your collection, Research Rabbit initiates the process of generating suggested additions. With each additional paper added, the accuracy of these recommendations gradually improves.

The recommendation system works somewhat like Spotify; by providing more information, i.e., by adding more papers, the app will get a better idea of what exactly the keyworded terms you’re interested in are and further help you discover research you may not have been aware was related to your work. This works especially well when you’re fairly unfamiliar with a specific topic by following the “bread-crumbs” based on a few papers that you’re interested in.

The main way to discover works related to the papers in your specific collection is by using the explore functions:


png depicting the file structure of the course template repository

Similar works

If you’re looking for papers that are related to your collection, you simply click the similar work button and a new literature map will fit into the original from your collection. Works from your collection will be displayed in green, while works outside your collection will be displayed in blue. By hovering over the elements, you can see their respective connections, the author’s (first or last author) names, and the year of publication.


png depicting the file structure of the course template repository

Exploring citations

Further clicking on one of the similar works will open a second sidebar containing more specific information on the paper in question, e.g., the title, all author’s names, and the abstract. But we’re not stopping there, as we can now also explore works related to the paper in question, e.g., by looking at all papers that have cited this specific paper using the all citations buttons. This, for example, reveals that the paper in the screenshot is apparently drawing information from different fields of research as the literature map is distributed into multiple clusters of separate maps.


png depicting the file structure of the course template repository

Down the Rabbit hole

As the name suggests, you can suggest this process immediately (probably not a good idea) and find different rabbit holes in the literature.

By selecting the help button on the right-hand side, you’ll open the help window. Here you’ll find frequently asked questions (ResearchRabbit FAQ), e.g., where does the data come from or how do I read the maps, as well as a youtube playlist of community created in-depth content and the Feature Overview tour, which will help you get started with ResearchRabbit.



Meta-analysis tools

For the neuroscientifically inclined, there is another way to find relevant literature by using web-based platforms that provide tools for automated meta-analysis of neuroimaging studies.

png depicting the file structure of the course template repository

These tools tend to use natural language processing (NLP) techniques to extract information from published studies and generate statistical summaries of brain activity patterns across different cognitive processes and experimental conditions. These summaries, called “maps,” can be used to identify brain regions that are commonly activated or deactivated across multiple studies and to generate hypotheses about the functional organization of the brain. Neurosynth can be used to, e.g., inform the selection of brain regions for further investigation, but also provides a weighted list of the contribution of separate papers, each with a separate brain map. Following, you can use these papers to base further meta-anylsis on the platform or add them to your collection in, e.g., ResearchRabbit.

You can further export both graphs and the list of papers and include them in your documentation/research.

Two platforms for this are:

Neuroquery

Neurosyth


Evaluate/Read a Scientific Paper

Of course, an essential part of evaluating literature is evaluating the quality of a paper itself.

Some overall tips would be:

  1. First, skim the article and identify its structure

  2. Distinguish main points/graphs

  3. Take notes/annotate along the way

  4. Read carefully and re-evaluate your annotations

The following is an in-depth guide on how to seriously engage with a scientific paper to gain the maximum information and benefit from it. This may seem discouraging due to the sheer scope of the task, but it’s a great idea to train and ultimately automatize this process.

The main takeaway from the following section is the six questions you should ask yourself about every paper to engage more deeply with the content and help kickstart your memory.

Ask six questions (Carey et al., 2020)

Regarding the entire work, including all sections, ask yourself the following questions:

  1. What do the author(s) want to know (motivation)?

  2. What did they do (approach/methods)?

  3. Why was it done that way (context within the field)?

  4. What do the results show (figures and data tables)?

  5. How did the author(s) interpret the results (interpretation/discussion)?

  6. What should be done next?

Of course, it’s not necessary to engage this deeply with every paper you come across. Don’t waste your time on papers that don’t relate to what you want to find out. In this case, concentrate on the first three steps presented below to evaluate the worth of a given paper for your work.

Ten simple rules for reading a scientific paper (Carey et al., 2020)

Rule 1: Pick your reading goal

Rule 2: Understand the author’s goal

Rule 3: Ask six questions

Rule 4: Unpack each figure and table

Rule 5: Understand the formatting intentions

Rule 6: Be critical

Rule 7: Be kind

Rule 8: Be ready to go the extra mile

Rule 9: Talk about it

Rule 10: Build on it

Rule 1: Pick your reading goal

The motivation to read a paper and the desired outcome (can) influence how one reads a paper and different priorities for different desired outcomes. Do you want a complete overview of the paper, or are you only interested in, e.g., the methods used?

Evaluate your goal and adapt your reading accordingly (e.g., skim instead of read in detail, etc.)

Rule 2: Understand the author’s goal

Ask yourself:

Why did the authors want to share a given study?

  • scientific field, scientific interest, author’s research

  • helps with interpreting data & understanding authors’ interpretation

In what form is the information presented?

  • type of article: methods, commentary, resources, research, review, etc.

  • formatting & content

  • intended purpose further shapes understanding of the author’s goal

Rule 3: Ask six questions

Regarding the entire work, including all sections, ask yourself the following questions:

  1. What do the author(s) want to know (motivation)?

  2. What did they do (approach/methods)?

  3. Why was it done that way (context within the field)?

  4. What do the results show (figures and data tables)?

  5. How did the author(s) interpret the results (interpretation/discussion)?

  6. What should be done next?

Rule 4: Unpack each figure and table

Skim/evaluate figures and tables before actually reading the paper. Evaluate:

  • intelligibility, complexity

  • x- and y-axes, color scheme, statistical approach (if one was used), the particular plotting approach

For each table containing data, evaluate:

  • intelligibility, complexity

  • experimental groups variables

  • presented statistics

Think about six questions of rule 3 and formulate the critical outcome/take-home message!

Rule 5: Understand the formatting intentions

There are distinct motivations and content for the distinct sections of a paper as discussed above. These may be further influenced by article type, journal policies, etc. but usually remain the same for every scientific paper.

Depending on what are you looking for, check different sections:

  • Overview of the results? → Abstract, Conclusion, Figures

  • Method? → Methods, Supplementary material

  • Results? → Methods, Results, Tables, Figures

  • Interpretation → Discussion

  • Overview of the literature? → Introduction

Rule 6: Be critical

Test the strength of conclusion by critically evaluating everything: hypotheses, methods, results, interpretation

  • self-fulfilling prophecies & expectations?

    • assumptions about data, results & interpretation

    • alignment of behavior

    • alternative hypothesis

Evaluate the paper in regards to open science practice and how it deals with the replicability/reproducibility crisis, publication bias of prior literature, QRP (questionable research practices, etc.).

  • use the Replicability Index, which provides a number of tools and information on how to spot problematic trends in published research.

  • critically evaluate the reliability of results

Rule 7: Be kind

  • If possible: Give the benefit of the doubt, most folks try to give their best

  • Minor things (typos, reference errors, certain visualizations, etc.) shouldn’t guide/influence evaluation of data, results, interpretation

  • make your critique about facts/data and not beliefs

  • be constructive and objective

Rule 8: Be ready to go the extra mile

Don’t expect everything necessary to fully understand a given paper to be present in it (unfortunately, most papers are not written for people new to the field):

  • lookup terms, definitions, models, etc.

  • consult cited references and prior work

  • evaluate supplementary materials (also check out the author’s other online presences, as e.g. some papers come with in-depth publications, e.g. https://oreoni.github.io/index.html)

Potentially read a paper more than once, each time with different goals (see rule 1): overview, understanding, evaluation, etc.

Annotate, annotate, annotate:

  • mark questions, unclear paragraphs, connections between figures, etc.

  • Share/create annotations with others (e.g., via google docs, etc.)

Rule 9: Talk about it

Prepare and engage with articles:

  • attend journal clubs

  • give paper presentations to peers

Discuss work with your colleagues, mentors, friends, families, etc. They will provide:

  • different points of view

  • different levels of discussion

  • different foci

Check open discussions, e.g., on Twitter/X.

Rule 10: Build on it

Think about the bigger picture:

  • how does the paper fit within current research and prior research work

  • situate a paper regarding your existing knowledge & new insights

  • use everything to inform your own research (i.e. think outside your own discipline)

Think about the paper and prior research work as building blocks that together create knowledge and the basis for further research.

Think about how aspects of the paper (methods, sample, etc.) can be integrated into your research.


Finding data

Gregory et al. (2018) provide Eleven Quick Tips for Finding Research Data”. Well, go through the most important tips in the following section and list some considerations of our own, but for more in-depth info, check out the paper itself.

Tip 1: Think about the data you need and why you need them.

Evaluate whether you seek data as the basis for a new study, for comparison or validation with existing results/data, or to simply explore or simulate the behavior or characteristics of certain types of data.

Next, make a list of the characteristics your data should fulfill for the above-identified purpose.

Requirements could include:

  • task/process or subject in question

  • data format (e.g., behavioral, questionnaire, EEG, fMRI, etc.)

  • spatial or temporal coverage (location/region and year or age range, etc.)

  • availability (free, “upon request”)

Tip 2: Select the most appropriate resource.

While you can find data available online, e.g., in:

Open Science Repositories: Repositories such as OSF, OpenNeuro, the OpenfMRI database, and the IEEE DataPort provide open access to MRI and EEG datasets, as well as other neuroimaging data.

Research Data Repositories: Zenodo, Figshare, and other research data repositories allow scientists to store, share, and publish their data in an open and transparent manner. These repositories are often committed to open access principles and provide a centralized location for data and metadata, as well as version control and preservation features.

Or explore databases specific to your field of study: For neuroscience, the most prominent would be:

You can further consult research data search engines:

Research data search engines help researchers discover and access (ideally high-quality) data repositories across different scientific disciplines.

One such service is FAIRsharing:

Fairsharing platform aims to promote the principles of Findability, Accessibility, Interoperability, and Reusability (FAIR) in research data management by providing a curated collection of resources that comply with these standards. With FAIRsharing, researchers can search for datasets, ontologies, and other research resources based on their scientific domain, data type, and other relevant parameters. By facilitating the discovery and reuse of research data, FAIRsharing aims to foster collaboration, accelerate scientific progress, and maximize the impact of research outcomes.

Other research data search engines are, for example:

A good starting point to find databases and search engines for your field of study is the constantly updated Wikipedia List of academic databases and search engines and List of online databases.

Tip 3: Construct your query strategically.

The same strategies as for literature search apply; make sure to explore multiple combinations of search terms and operators during your search and document every search query. Additionally, use multiple search engines/databases and tailor your search queries and vocabulary for each individual database.

Tip 4: Make the repository work for you.

Make yourself familiar with the platform and investigate further resources on how to best work with the search engine/database. For example, follow the Openneuro User Guide to make the most out of the platform.

Tip 6: Assess data relevance and fitness -for -use.

Cross-reference with the requirements provided in Tip 1. Also, evaluate the data format (e.g., the BIDS standard] and check the quality parameters that most platforms include. If a dataset looks fit for your purposes, investigate further by downloading samples and implementing your own systematic quality assessment or analyzing the descriptive statistics provided with the dataset. This is not a trivial task, so make sure to search for relevant tools and papers on the subject and consult with your colleagues or advisors.

Tip 7: Save your search and data source details.

Documentation!! Record your search queries, ideally on which date you searched, as well as the name of the dataset in question and its persistent identifier (digital object identifier (DOI); Global et al. (GUID).

Tip 8: Look for data services, not just data.

Data may not necessarily be available without using a specific service and can only be accessed via an application programming interface (API).

Examples of such services include for example:

NSW Climate Data Portal

Google Earth Engine

Amazon Web Services (AWS) public datasets

Tip 9: Monitor the latest data.

New publications have, at times, the requirement to host their data on public platforms; some platforms provide the ability to set keyword-based e-mail alarms to keep you up to date. Otherwise, monitor recent publications (e.g., as shown above for Google Scholar) and check back with the databases.

Tip 10: Treat sensitive data responsibly.

This speaks for itself; most data that you’ll find online is anonymized and cleared for the public, but if you ever work with data from colleagues or collect your own data, you are required to apply to local data protection laws, especially when dealing with identifiable data of human subjects, for more info see our Lecture on data management .

Tip 11: Give back (cite and share data).

Attribute: Identify how the authors of a dataset would like to be cited and apply this standard wherever possible

Give Feedback: Offer input to the creators of the data or the data repository regarding any problems related to the accessibility of the data, the quality of the data, or the completeness and interpretability of metadata (i.e., data describing the dataset, such as descriptive population statistics).

Make your own data open & accessible: For a guide on how to best do that, see our Lecture on data management .

References

Bolzan, J. A., & de Oliveira, C. L. A compact guide to the systematic review and meta-analysis of the literature in neuroscience. Journal for Reproducibility in Neuroscience, 2, https://doi.org/10.31885/jrn.2.2021.1669 (2021)

Carey, M. A., Steiner, K. L., & Petri Jr, W. A. (2020). Ten simple rules for reading a scientific paper. PLoS computational biology, 16(7), e1008032.

Gregory, K., Khalsa, S. J., Michener, W. K., Psomopoulos, F. E., De Waard, A., & Wu, M. (2018). Eleven quick tips for finding research data. PLoS Computational Biology, 14(4), e1006038.

Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. K. (2019). Best practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses. Research Synthesis Methods, 10(3), 330-342.

Pigott, T. D. & Polanin, J. R. Methodological Guidance Paper: High-Quality Meta-Analysis in a Systematic Review. Review of Educational Research 90, 24–46 https://doi.org/10.3102/0034654319877153 (2020).

Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane, 2022. Available from www.training.cochrane.org/handbook.

Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012, January). Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT international health informatics symposium (pp. 819-824).

previous

Project Design in the Digital Age

next

Scientific Writing & Digital Tools