Open Access. Powered by Scholars. Published by Universities.®
Articles 1 - 15 of 15
Full-Text Articles in Computer Sciences
Can You Summarize This? Identifying Correlates Of Input Difficulty For Generic Multi-Document Summarization, Ani Nenkova, Annie Louis
Can You Summarize This? Identifying Correlates Of Input Difficulty For Generic Multi-Document Summarization, Ani Nenkova, Annie Louis
Ani Nenkova
Different summarization requirements could make the writing of a good summarymore difficult, or easier. Summary length and the characteristics of the input are such constraints influencing the quality of a potential summary. In this paper we report the results of a quantitative analysis on data from large-scale evaluations of multi-document summarization, empirically confirming this hypothesis. We further show that features measuring the cohesiveness of the input are highly correlated with eventual summary quality and that it is possible to use these as features to predict the difficulty of new, unseen, summarization inputs.
Discourse Indicators For Content Selection In Summaization, Annie Louis, Aravind K. Joshi, Ani Nenkova
Discourse Indicators For Content Selection In Summaization, Annie Louis, Aravind K. Joshi, Ani Nenkova
Ani Nenkova
We present analyses aimed at eliciting which specific aspects of discourse provide the strongest indication for text importance. In the context of content selection for single document summarization of news, we examine the benefits of both the graph structure of text provided by discourse relations and the semantic sense of these relations. We find that structure information is the most robust indicator of importance. Semantic sense only provides constraints on content selection but is not indicative of important content by itself. However, sense features complement structure information and lead to improved performance. Further, both types of discourse information prove complementary …
Automatic Sense Prediction For Implicit Discourse Relations In Text, Emily Pitler, Annie Louis, Ani Nenkova
Automatic Sense Prediction For Implicit Discourse Relations In Text, Emily Pitler, Annie Louis, Ani Nenkova
Ani Nenkova
We present a series of experiments on automatically identifying the sense of implicit discourse relations, i.e. relations that are not marked with a discourse connective such as “but” or “because”. We work with a corpus of implicit relations present in newspaper text and report results on a test set that is representative of the naturally occurring distribution of senses. We use several linguistically informed features, including polarity tags, Levin verb classes, length of verb phrases, modality, context, and lexical features. In addition, we revisit past approaches using lexical pairs from unannotated text as features, explain some of their shortcomings and …
Automatic Evaluation Of Linguistic Quality In Multi-Document Summarization, Emily Pitler, Annie Louis, Ani Nenkova
Automatic Evaluation Of Linguistic Quality In Multi-Document Summarization, Emily Pitler, Annie Louis, Ani Nenkova
Ani Nenkova
To date, few attempts have been made to develop and validate methods for automatic evaluation of linguistic quality in text summarization. We present the first systematic assessment of several diverse classes of metrics designed to capture various aspects of well-written text. We train and test linguistic quality models on consecutive years of NIST evaluation data in order to show the generality of results. For grammaticality, the best results come from a set of syntactic features. Focus, coherence and referential clarity are best evaluated by a class of features measuring local coherence on the basis of cosine similarity between sentences, coreference …
Structural Features For Predicting The Linguistic Quality Of Text: Applications To Machine Translation, Automatic Summarization And Human-Authored Text, Ani Nenkova, Jieun Chae, Annie Louis, Emily Pitler
Structural Features For Predicting The Linguistic Quality Of Text: Applications To Machine Translation, Automatic Summarization And Human-Authored Text, Ani Nenkova, Jieun Chae, Annie Louis, Emily Pitler
Ani Nenkova
Sentence structure is considered to be an important component of the overall linguistic quality of text. Yet few empirical studies have sought to characterize how and to what extent structural features determine fluency and linguistic quality. We report the results of experiments on the predictive power of syntactic phrasing statistics and other structural features for these aspects of text. Manual assessments of sentence fluency for machine translation evaluation and text quality for summarization evaluation are used as gold-standard. We find that many structural features related to phrase length are weakly but significantly correlated with fluency and classifiers based on the …
Measuring Importance And Query Relevance In Toopic-Focused Multi-Document Summarization, Surabhi Gupta, Ani Nenkova, Dan Jurafsky
Measuring Importance And Query Relevance In Toopic-Focused Multi-Document Summarization, Surabhi Gupta, Ani Nenkova, Dan Jurafsky
Ani Nenkova
The increasing complexity of summarization systems makes it difficult to analyze exactly which modules make a difference in performance. We carried out a principled comparison between the two most commonly used schemes for assigning importance to words in the context of query focused multi-document summarization: raw frequency (word probability) and log-likelihood ratio. We demonstrate that the advantages of log-likelihood ratio come from its known distributional properties which allow for the identification of a set of words that in its entirety defines the aboutness of the input. We also find that LLR is more suitable for query-focused summarization since, unlike raw …
Using Entity Features To Classify Implicit Discourse Relations, Annie Louis, Aravind K. Joshi, Rashmi Prasad, Ani Nenkova
Using Entity Features To Classify Implicit Discourse Relations, Annie Louis, Aravind K. Joshi, Rashmi Prasad, Ani Nenkova
Ani Nenkova
We report results on predicting the sense of implicit discourse relations between adjacent sentences in text. Our investigation concentrates on the association between discourse relations and properties of the referring expressions that appear in the related sentences. The properties of interest include coreference information, grammatical role, information status and syntactic form of referring expressions. Predicting the sense of implicit discourse relations based on these features is considerably better than a random baseline and several of the most discriminative features conform with linguistic intuitions. However, these features do not perform as well as lexical features traditionally used for sense prediction.
Creating Local Coherence: An Empirical Assessment, Annie Louis, Ani Nenkova
Creating Local Coherence: An Empirical Assessment, Annie Louis, Ani Nenkova
Ani Nenkova
Two of the mechanisms for creating natural transitions between adjacent sentences in a text, resulting in local coherence, involve discourse relations and switches of focus of attention between discourse entities. These two aspects of local coherence have been traditionally discussed and studied separately. But some empirical studies have given strong evidence for the necessity of understanding how the two types of coherence-creating devices interact. Here we present a joint corpus study of discourse relations and entity coherence exhibited in news texts from the Wall Street Journal and test several hypotheses expressed in earlier work about their interaction.
Automatically Evaluating Content Selection In Summarization Without Human Models, Annie Louis, Ani Nenkova
Automatically Evaluating Content Selection In Summarization Without Human Models, Annie Louis, Ani Nenkova
Ani Nenkova
We present a fully automatic method for content selection evaluation in summarization that does not require the creation of human model summaries. Our work capitalizes on the assumption that the distribution of words in the input and an informative summary of that input should be similar to each other. Results on a large scale evaluation from the Text Analysis Conference show that input-summary comparisons are very effective for the evaluation of content selection. Our automatic methods rank participating systems similarly to manual model-based pyramid evaluation and to manual human judgments of responsiveness. The best feature, Jensen- Shannon divergence, leads to …
Entity-Driven Rewrite For Multi-Document Summarization, Ani Nenkova
Entity-Driven Rewrite For Multi-Document Summarization, Ani Nenkova
Ani Nenkova
In this paper we explore the benefits from and shortcomings of entity-driven noun phrase rewriting for multi-document summarization of news. The approach leads to 20% to 50% different content in the summary in comparison to an extractive summary produced using the same underlying approach, showing the promise the technique has to offer. In addition, summaries produced using entity-driven rewrite have higher linguistic quality than a comparison non-extractive system. Some improvement is also seen in content selection over extractive summarization as measured by pyramid method evaluation.
Detecting Prominence In Conversational Speech: Pitch Accent, Givenness And Focus, Vivek Kumar Rangarajan Sridhar, Ani Nenkova, Shrikanth Narayanan, Dan Jurafsky
Detecting Prominence In Conversational Speech: Pitch Accent, Givenness And Focus, Vivek Kumar Rangarajan Sridhar, Ani Nenkova, Shrikanth Narayanan, Dan Jurafsky
Ani Nenkova
The variability and reduction that are characteristic of talking in natural interaction make it very difficult to detect prominence in conversational speech. In this paper, we present analytic studies and automatic detection results for pitch accent, as well as on the realization of information structure phenomena like givenness and focus. For pitch accent, our conditional random field model combining acoustic and textual features has an accuracy of 78%, substantially better than chance performance of 58%. For givenness and focus, our analysis demonstrates that even in conversational speech there are measurable differences in acoustic properties and that an automatic detector for …
Modelling Prominence And Emphasis Improves Unit-Selection Synthesis, Volker Strom, Ani Nenkova, Robert Clark, Yolanda Vazquez-Alvarez, Jason Brenier, Simon King, Dan Jurafsky
Modelling Prominence And Emphasis Improves Unit-Selection Synthesis, Volker Strom, Ani Nenkova, Robert Clark, Yolanda Vazquez-Alvarez, Jason Brenier, Simon King, Dan Jurafsky
Ani Nenkova
We describe the results of large scale perception experiments showing improvements in synthesising two distinct kinds of prominence: standard pitch-accent and strong emphatic accents. Previously prominence assignment has been mainly evaluated by computing accuracy on a prominence-labelled test set. By contrast we integrated an automatic pitch-accent classifier into the unit selection target cost and showed that listeners preferred these synthesised sentences. We also describe an improved recording script for collecting emphatic accents, and show that generating emphatic accents leads to further improvements in the fiction genre over incorporating pitch accent only. Finally, we show differences in the effects of prominence …
Predicting The Fluency Of Text With Shallow Structural Features: Case Studies Of Machine Tanslation And Human-Written Text, Jieun Chae, Ani Nenkova
Predicting The Fluency Of Text With Shallow Structural Features: Case Studies Of Machine Tanslation And Human-Written Text, Jieun Chae, Ani Nenkova
Ani Nenkova
Sentence fluency is an important component of overall text readability but few studies in natural language processing have sought to understand the factors that define it. We report the results of an initial study into the predictive power of surface syntactic statistics for the task; we use fluency assessments done for the purpose of evaluating machine translation. We find that these features are weakly but significantly correlated with fluency. Machine and human translations can be distinguished with accuracy over 80%. The performance of pairwise comparison of fluency is also very high—over 90% for a multi-layer perceptron classifier. We also test …
Class-Level Spectral Features For Emotion Recognition, Dmitri Bitouk, Ragini Verma, Ani Nenkova
Class-Level Spectral Features For Emotion Recognition, Dmitri Bitouk, Ragini Verma, Ani Nenkova
Ani Nenkova
The most common approaches to automatic emotion recognition rely on utterance-level prosodic features. Recent studies have shown that utterance-level statistics of segmental spectral features also contain rich information about expressivity and emotion. In our work we introduce a more fine-grained yet robust set of spectral features: statistics of Mel-Frequency Cepstral Coefficients computed over three phoneme type classes of interest – stressed vowels, unstressed vowels and consonants in the utterance. We investigate performance of our features in the task of speaker-independent emotion recognition using two publicly available datasets. Our experimental results clearly indicate that indeed both the richer set of spectral …
Performance Confidence Estimation For Automatic Summarization, Annie Louis, Ani Nenkova
Performance Confidence Estimation For Automatic Summarization, Annie Louis, Ani Nenkova
Ani Nenkova
We address the task of automatically predicting if summarization system performance will be good or bad based on features derived directly from either single- or multi-document inputs. Our labelled corpus for the task is composed of data from large scale evaluations completed over the span of several years. The variation of data between years allows for a comprehensive analysis of the robustness of features, but poses a challenge for building a combined corpus which can be used for training and testing. Still, we find that the problem can be mitigated by appropriately normalizing for differences within each year. We examine …