Not so weak-PICO: Leveraging weak supervision for Participants, Interventions, and Outcomes recognition for systematic review automation - Dhrangadhariya, Anjani; Müller, Henning (Dryad, 2022)
Objective: PICO (Participants, Interventions, Comparators, Outcomes)
analysis is vital but time-consuming for conducting systematic reviews
(SRs). Supervised machine learning can help fully automate it, but a lack
of large annotated corpora limits the quality of automated PICO
recognition systems. The largest currently available PICO corpus is
manually annotated, which is an approach that is often too expensive for
the scientific community to apply. Depending on the specific SR question,
PICO criteria are extended to PICOC (C-Context), PICOT (T-timeframe), and
PIBOSO (B-Background, S-Study design, O-Other) meaning the static
hand-labelled corpora need to undergo costly re-annotation as per the
downstream requirements. We aim to test the feasibility of designing a
weak supervision system to extract these entities without hand-labelled
data. Methodology: We decompose PICO spans into its constituent entities
and re-purpose multiple medical and non-medical ontologies and
expert-generated rules to obtain multiple noisy labels for these entities.
These labels obtained using several sources are then aggregated using
simple majority voting and generative modelling approaches. The resulting
programmatic labels are used as weak signals to train a weakly-supervised
discriminative model and observe performance changes. We explore mistakes
in the currently available PICO corpus that could have led to inaccurate
evaluation of several automation methods. Results: We present Weak-PICO, a
weakly-supervised PICO entity recognition approach using medical and
non-medical ontologies, dictionaries and expert-generated rules. Our
approach does not use hand-labelled data. Conclusion: Weak supervision
using weak-PICO for PICO entity recognition has encouraging results, and
the approach can potentially extend to more clinical entities readily.
This upload contains four main zip files. ds_cto_dict.zip: This zip file
contains the four distant supervision dictionaries (P: participant.txt, I
= intervention.txt, intervetion_syn.txt, O: outcome.txt) generated from
clinicaltrials.gov using the Methodology described in Distant-CTO
(https://aclanthology.org/2022.bionlp-1.34/). These dictionaries were used
to create distant supervision labelling functions as described in the
Labelling sources subsection of the Methodology. The data was derived from
https://clinicaltrials.gov/ handcrafted_dictionaries.zip: This zip folder
contains three files 1) gender_sexuality.txt: a list of possible genders
and sexual orientations found across the web. The list needs to be more
comprehensive. 2) endpoints_dict.txt: contains outcome names and the names
of questionnaires used to measure outcomes assembled from PROM
questionnaires and PROMs. and 3) comparator_dict: contains a list of
idiosyncratic comparator terms like a sham, saline, placebo, etc.,
compiled from the literature search. The list needs to be more
comprehensive. test_ebm_correctedlabels.tsv: EBM-PICO is a widely used
dataset with PICO annotations at two levels: span-level or coarse-grained
and entity-level or fine-grained. Span-level annotations encompass the
full information about each class. Entity-level annotations cover the more
fine-grained information at the entity level, with PICO classes further
divided into fine-grained subclasses. For example, the coarse-grained
Participant span is further divided into participant age, gender,
condition and sample size in the randomised controlled trial. This dataset
comes pre-divided into a training set (n=4,933) annotated through
crowd-sourcing and an expert annotated gold test set (n=191) for
evaluation. The EBM-PICO annotation guidelines caution about
variable annotation quality. Abaho et al. developed a
framework to post-hoc correct EBM-PICO outcomes annotation
inconsistencies. Lee et al. studied annotation span disagreements
suggesting variability across the annotators. Low annotation quality in
the training dataset is excusable, but the errors in the test set can lead
to faulty evaluation of the downstream ML methods. We evaluate 1% of the
EBM-PICO training set tokens to gauge the possible reasons for the
fine-grained labelling errors and use this exercise to conduct an
error-focused PICO re-annotation for the EBM-PICO gold test set. The file
'test_ebm_correctedlabels.tsv' has error corrected EBM-PICO gold
test set. This dataset could be used as a complementary evalution set
along with EBM-PICO test set. error_analysis.zip: This .zip file contains
three .tsv files for each PICO class to identify possible errors in about
1% (about 12,962 tokens) of the EBM-PICO training set.
All the datasets could be opened using text editors or Google sheets. The
.zip files in the dataset can be opened using the archive utility on Mac
OS and unzip functionality in Linux. (All Windows and Apple operating
systems support the use of ZIP files without additional third-party
software)
Keywords: FOS: Computer and information sciences; error analysis; corpus creation; Information extraction; Deep learning; Natural language processing; Systematic reviews; Evidence based medicine