Optimized Machine Learning Methods Predict Discourse Segment Type in Biological Research Articles

Abstract

To define salient rhetorical elements in scholarly text, we have earlier defined a set of Discourse Segment Types: semantically defined spans of discourse at the level of a clause with a single rhetorical purpose, such as Hypothesis, Method or Result. In this paper, we use machine learning methods to predict these Discourse Segment Types in a corpus of biomedical research papers. The initial experiment used features related to verb type and form, obtaining F-scores ranging from 0.41–0.65. To improve our results, we explored a variety of methods for balancing classes, before applying classification algorithms. We also performed an ablation study and stepwise approach for feature selection. Through these feature selection processes, we were able to reduce our 37 features to the 9 most informative ones, while maintaining F1 scores in the range of 0.63–0.65. Next, we performed an experiment with a reduced set of target classes. Using only verb tense features, logistic regression, a decision tree classifier and a random forest classifier, we predicted that a segment type was either a Result/Method or a Fact/Implication, with F1 scores above 0.8. Interestingly, findings from this machine learning approach are in line with a reader experiment, which found a correlation between verb tense and a biomedical reader’s interpretation of discourse segment type. This suggests that experimental and concept-centric discourse in biology texts can be distinguished by humans or machines, using verb tense as a key feature.

Appendices

Appendices

1.1 Appendix 5.1. Starting Feature List and Descriptions

Feature class	Feature	Included in experiment #
Frequently Used Verb	Top 10 Verb	1
Frequently Used Verb	‘Show’ Verb	1, 3
Verb Tense	Future	1, 3, 4
Verb Tense	Gerund	1, 4
Verb Tense	Past	1, 2, 3, 4
Verb Tense	Past participle	1, 4
Verb Tense	Past perfect	1, 4
Verb Tense	Past progressive	1, 3, 4
Verb Tense	Present	1, 2, 4
Verb Tense	Present perfect	1, 4
Verb Tense	Present progressive	1, 4
Verb Tense	To-infinitive	1, 2, 3, 4
Verb Class	Cause and effect	1
Verb Class	Change and growth	1
Verb Class	Discourse verb	1
Verb Class	Interpretation	1, 2, 3
Verb Class	Investigation	1, 2, 3
Verb Class	None	1
Verb Class	Observation	1, 3
Verb Class	Prediction	1
Verb Class	Procedure	1, 2, 3
Verb Class	Properties	1, 3
Modality Marker	Modal	1, 2, 3
Modality Marker	Verb class interpretation	1, 2, 3
Modality Marker	Ruled by verb class interpretation	1, 2, 3
Modality Marker	Reference internal	1
Modality Marker	Reference external	1
Modality Marker	First person	1
Modality Marker	Modal significant_ly	1
Modality Marker	Possible_ility_ly	1
Modality Marker	Potential_ly	1
Modality Marker	UN_Likely	1
Modality Marker	Sum_Adverbs_YesNO	1

1.2 Appendix 5.2. Description of Sampling Methods Used

Sampling method	Description	Method
RandomUnderSampler	Undersamples the majority classes by randomly picking samples	Undersampler
Tomeklinks	Undersamples the majority classes by removing Tomek’s links	Undersampler
ClusterCentroids	Under samples the majority classes by replacing a cluster of the majority samples by the cluster centroid of a KMeans algorithm	Undersampler
CondensedNearestNeighbor	Under samples the majority classes using the condensed nearest neighbor method	Undersampler
OneSidedSelection	Uses one-sided selection method on majority classes	Undersampler
InstanceHardnessThreshold	Samples with lower probabilities are removed from the majority class	Undersampler
RandomOverSampler	Randomly generates new samples from the minority classes	Oversampler
SMOTE	Synthetic Minority Oversampling Technique; generates new samples of minority class by interpolation	Oversampler
SMOTEborderline	Generates new samples of minority class specific to the borders between two classes	Oversampler
SMOTEborderline2	Generates new samples of minority class specific to the borders between two classes	Oversampler
SMOTETomek	Combines use of SMOTE on minority class and Tomek Links on majority class	Over and undersampler
SMOTEENN	Combines use of SMOTE on minority class and Edited Nearest Neighbors on majority class	Over and undersampler

1.3 Appendix 5.3. Accuracy, Precision, Recall and F1 Scores of All 36 Models Tested

Classifier	Class balancer	Accuracy	Precision	Recall	F1
LR	No Class Balancer	0.62	0.68	0.63	0.64
DTC	No Class Balancer	0.64	0.64	0.64	0.64
RFC	No Class Balancer	0.64	0.65	0.65	0.64
LR	RandomUnderSampler	0.58	0.64	0.58	0.59
DTC	RandomUnderSampler	0.55	0.64	0.55	0.56
RFC	RandomUnderSampler	0.57	0.63	0.56	0.57
LR	Tomeklinks	0.63	0.68	0.63	0.64
DTC	Tomeklinks	0.64	0.64	0.64	0.64
RFC	Tomeklinks	0.64	0.64	0.64	0.64
LR	ClusterCentroids	0.55	0.64	0.55	0.55
DTC	ClusterCentroids	0.35	0.48	0.35	0.32
RFC	ClusterCentroids	0.38	0.47	0.38	0.35
LR	CondensedNearestNeighbor	0.62	0.67	0.62	0.62
DTC	CondensedNearestNeighbor	0.53	0.59	0.53	0.53
RFC	CondensedNearestNeighbor	0.55	0.60	0.55	0.55
LR	OneSidedSelection	0.60	0.65	0.6	0.61
DTC	OneSidedSelection	0.47	0.47	0.47	0.46
RFC	OneSidedSelection	0.48	0.43	0.48	0.45
LR	InstanceHarnessThreshold	0.46	0.58	0.46	0.5
DTC	InstanceHarnessThreshold	0.37	0.61	0.37	0.41
RFC	InstanceHarnessThreshold	0.40	0.61	0.4	0.44
LR	RandomOverSampler	0.63	0.68	0.63	0.64
DTC	RandomOverSampler	0.60	0.64	0.6	0.61
RFC	RandomOverSampler	0.61	0.64	0.61	0.61
LR	SMOTE	0.63	0.68	0.63	0.64
DTC	SMOTE	0.62	0.64	0.63	0.63
RFC	SMOTE	0.63	0.64	0.63	0.63
LR	SMOTEborderline	0.63	0.68	0.63	0.65
DTC	SMOTEborderline	0.63	0.64	0.63	0.63
RFC	SMOTEborderline	0.62	0.63	0.62	0.62
LR	SMOTEborderline2	0.63	0.68	0.63	0.64
DTC	SMOTEborderline3	0.63	0.64	0.63	0.63
RFC	SMOTEborderline4	0.62	0.64	0.62	0.62
LR	SMOTETomek	0.63	0.68	0.63	0.65
DTC	SMOTETomek	0.63	0.64	0.63	0.63
RFC	SMOTETomek	0.63	0.65	0.63	0.63
LR	SMOTEENN	0.50	0.63	0.50	0.52
DTC	SMOTEENN	0.42	0.65	0.42	0.45
RFC	SMOTEENN	0.44	0.63	0.44	0.46