Sentiment and Opinion Analysis
In the opinion mining shared task we want to address problems such as determining text subjectivity and polarity, and sentiment analysis. Although these problems have been already approached from different perspectives, most of the research has been carried out on specific domain data and applications where users are requested to rate services or products. Our intention is to focus the attention into the more general domain in which Web 2.0 users express their sentiments and opinions in their daily interaction within a virtual community.
Motivation:
The web 2.0 is transferring the role of content generation from traditional editorial sources to the general public. In this new context, common citizens are able to express their sentiments and opinions about a wide range of elements, facts and events within their own local community, as well as in the global one. In this way, general public becomes both the informant and the informed. It is expected that the collective intelligence and knowledge available through user generated contents in web 2.0 will constitute an important modulator of the social, economic and politic spheres of the information society.
Objectives:
The main objective of the present shared task is to organize opinion mining and sentiment analysis evaluations in the open context of web 2.0. Instead of restricting the analysis to a specific domain and/or subject oriented application, the goal of this evaluation is to deal with both problems in a general an open domain.
Description:
This shared task is organized in two tracks: sentiment analysis and opinion analysis. For both tracks, the common shared-task training dataset should be used. This training dataset is described and can be downloaded from this link. Similarly, evaluation datasets, which are specific for each track, will be provided for evaluation purposes during the first week of March 2009.
Training data for both tracks should be restricted to the provided dataset, from which each participant team should filter and extract those subsets considered relevant for the specific approach to be implemented. Any possible approach is allowed: rule-based, statistical, supervised, unsupervised, etc. Any additional source of information and resources such as dictionaries, language models, annotated data (extracted from the provided dataset), etc. should be notified to and shared with other participant teams through the resource-sharing page
The main goal of the sentiment analysis track is to assign a given comment with a degree of association to four basic categories: neutral, happy, angry and sad. On the other hand, the main goal of the opinion analysis track is to assign a given comment with a degree of association to three basic categories: factual, opinionated-positive and opinionanted-negative. The degree of association for each category should be provided by means of a real value within the range between zero (not associated at all) to one (completely associated).
Evaluation:
In both tracks, sentiment analysis and opinion analysis, evaluation of system performances will be conducted over a test dataset specifically prepared for each track. System outputs should provide a degree of association to each basic category for each comment in the test data set. Evaluation scores for system outputs will be computed in terms of vector distances between each system output and its corresponding reference vector. Reference vectors will be computed from manual annotations of the test dataset, which will be made public after submission of the evaluation results.
