How transferable are the datasets collected by active learners?
Active learning is a widely-used training strategy for maximizing predictive performance subject to a fixed annotation budget. Between rounds of training, an active learner iteratively selects examples for annotation, typically based on some measure of the model's uncertainty, coupling the acquired dataset with the underlying model. However, owing to the high cost of annotation and the rapid pace of model development, labeled datasets may remain valuable long after a particular model is surpassed by new technology. In this paper, we investigate the transferability of datasets collected with an acquisition model A to a distinct successor model S. We seek to characterize whether the benefits of active learning persist when A and S are different models. To this end, we consider two standard NLP tasks and associated datasets: text classification and sequence tagging. We find that training S on a dataset actively acquired with a (different) model A typically yields worse performance than when S is trained with "native" data (i.e., acquired actively using S), and often performs worse than training on i.i.d. sampled data. These findings have implications for the use of active learning in practice,suggesting that it is better suited to cases where models are updated no more frequently than labeled data.
NurtureToken New!

Token crowdsale for this paper ends in

Buy Nurture Tokens

Authors

Are you an author of this paper? Check the Twitter handle we have for you is correct.

David Lowell (add twitter)
Zachary C. Lipton (edit)
Byron C. Wallace (edit)
Ask The Authors

Ask the authors of this paper a question or leave a comment.

Read it. Rate it.
#1. Which part of the paper did you read?

#2. The paper contains new data or analyses that is openly accessible?
#3. The conclusion is supported by the data and analyses?
#4. The conclusion is of scientific interest?
#5. The result is likely to lead to future research?

Github
Stargazers:
12
Forks:
1
Open Issues:
0
Network:
1
Subscribers:
3
Language:
Python
Bayesian Deep Active Learning for Natural Language Processing Tasks
Youtube
Link:
None (add)
Views:
0
Likes:
0
Dislikes:
0
Favorites:
0
Comments:
0
Other
Sample Sizes (N=):
Inserted:
Words Total:
Words Unique:
Source:
Abstract:
None
07/15/18 09:32PM
4,969
1,842
Tweets
d_i_j_k_stra: Practical Obstacles to Deploying Active Learning (Lowell et al https://t.co/2JDjfN1Je3 ) Studies different heuristics, acquisition models etc across datasets but fails to find a common pattern. Interestingly seq labeling is an outlier which active learning works really well.
zacharylipton: Active learning sounds great, but obstacles lurk: 1) data coupled precariously to models used for selection; 2) reported numbers are in hindsight; 3) brittle to seemingly irrelevant choices (e.g. word embeddings); & more...!! Accepted at #EMNLP2019 (1/3) https://t.co/8xSx4WfOE2
zacharylipton: Datasets' shelf-life often longer than models'. In *active learning*, dataset is coupled to the uncertainty of our model. What happens when we train a new model B, using data collected by model A? Results concerning. Work w David Lowell & @byron_c_wallace https://t.co/8xSx4WfOE2
arxiv_org: How transferable are the datasets collected by active learners?. https://t.co/3gwRFKabPk https://t.co/sy4c6NLRXw
Images
Related