terça-feira, 8 de abril de 2014

Having fun creating a JavaFX App to Visualize Tweet Sentiments

A cool think that people loves to do is to create apps that analyze the sentiment of some text stream, such as Twitter. To better understand, see a graphical sentiment analysis for the Java 8 release messages on Twitter (source):

In this post, I'm going to create a simple, really simple, app that queries twitter, and then access the Sentiment140 REST API to analyze the tweet's content sentiment. After this, I'll count the results in a chart and also list the text content in a table.

Using the Sentiment140 REST API

The API is really simple to use. Since I'm lazy, I love simplicity. To use it, submit a text and it will return a simple JSON with a number value called polarity, which possible values are:
  • 0: negative
  • 2: neutral
  • 4: positive
The API documentation contains all the information you need to use it. Again, I simply loved how easy is to use it. For example, if I want to submit the text "I Love REST APIs" to the API, I would have to HTTP POST a JSON containing it to the URL http://www.sentiment140.com/api/bulkClassifyJson  I would use the following curl command:

$ curl -d "{'data': [{'text': 'I Love REST APIs'}]}" http://www.sentiment140.com/api/bulkClassifyJson

And it will return:

{"data":[{"text":"I Love REST APIs","polarity":4,"meta":{"language":"en"}}]}

 See, it's not hard!

Querying Twitter

I wanted to query twitter, it was my plan... However, it's "new" API requires me to use OAuth to query. Twitter4j API, however, is great to handle Twitter, so I decided to keep twitter... See the code I use to query twitter:

private List searchTwitter(String q){ 
        Twitter twitter = new TwitterFactory().getInstance();
        Query twitterQuery = new Query("lang:en " + q); 
                return twitter.search(twitterQuery)
                        // map to text and replace all special chars
                        .map(s -> s.getText().replaceAll("[^a-zA-Z0-9\\s\\,\\.\\']+",""))
        }catch(Exception e){ 
                throw new RuntimeException(e);

Accessing the APIs with Java

The first thing to do is to use model what we want in a class and what is want is basically what the Sentiment140 API returns. So we created a Java bean named TextSentiment which code is below:

@JsonIgnoreProperties(ignoreUnknown = true)
public class TextSentiment{
        private String text;
        private int polarity;

         // getters setters and constructors

public class TextSentimentList{
        private List textSentiments;
         // getters setters 

Now, we have need to find a good way to send HTTP requests to this API and return instances of TextSentiment. To do this We created a class called TextSentimentService, and in this class we make use of JAX-RS 2 Client API to query Sentiment140, so we don't waste time parsing data... The method getSentiment140 was used to get the sentiment of all the Strings we passed

import org.jugvale.sentiments.model.TextSentimentList;
import org.jugvale.sentiments.model.TextSentiment;
import java.util.List;
import java.util.stream.Collectors;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.Entity;

public class TextSentimentService{

        private final String SENTIMENTS_BASE_URL = "http://www.sentiment140.com/api/bulkClassifyJson";
        private final String MEDIA_TYPE = "application/json";

        public TextSentimentList getSentiments(List texts){
                TextSentimentList requestEntity = new TextSentimentList();
                return ClientBuilder.newClient()
                                .buildPost(Entity.entity(requestEntity, MEDIA_TYPE))


To access Twitter, we created a class SearchService, which has a method called search. In this method, we specify the provider we are wanting to retrieve texts to analyze, in our case, we have twitter:

import java.util.List;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.client.Entity;
import twitter4j.*;
import java.util.stream.Collectors;
public class SearchService{

         *It's mine!!! Please, use yours
         * */

        private final static String TWITTER_CONSUMER_KEY = "{KEY}";
        private final static String TWITTER_CONSUMER_KEY_SECRET = "{SECRET}";

        public static enum Provider{

                private String name;
                Provider(String name){
                        this.name = name;
                public String toString(){
                        return this.name;

        public List search(String q, Provider p){ 
                switch (p){
                        case TWITTER:
                                return searchTwitter(q);
                                throw new IllegalArgumentException("Provider \"" + p + "\" not implemented");
        // searchTwitter method...

Notice that both Service classes can be extended to make use of another sentiment API or another social media API.

Testing our service classes

Before continuing with our application, let's make a simple test of the Sentiment Service API, see the code below:

import org.junit.Test;
import org.jugvale.sentiments.service.*;
import org.jugvale.sentiments.model.*;
import java.util.Arrays;

public class TestService{

        public void testServices(){
                String TXT_1 = "obama is awesome";
                String TXT_2 = "obama sucks";
                String TXT_3 = "obama is eating a potato";
                TextSentimentService s = new TextSentimentService();
                TextSentimentList list = s.getSentiments(Arrays.asList(TXT_1, TXT_2, TXT_3));
                //TODO: Create a Map with key equal the text and polarity is the value

With this small and simple test, we make sure we are accessing the API and returning something, so we also confirm the parsing of the response if correct.

The JavaFX APP

 Our small APP contains one text field, one chart and one table! The user enters the query on the TextField, press enter, then our app takes that query to search the social media, get the results and then it sends the texts to the Sentiment140 API analyze. With the results, we update the user interface. See the code:

import javafx.application.Application;
import javafx.scene.Scene;
import javafx.stage.Stage;
import javafx.collections.FXCollections;
import javafx.event.ActionEvent;
import javafx.scene.control.*;
import javafx.scene.control.cell.*;
import javafx.scene.layout.VBox;
import javafx.scene.chart.*;
import org.jugvale.sentiments.service.*;
import org.jugvale.sentiments.model.*;
import java.util.*;

public class App extends Application{

 final TextSentimentService textSentimentService = new TextSentimentService();
 final SearchService searchService = new SearchService();
 final PieChart chart =  new PieChart();
 final TableView table = new TableView<>();
 final TextField txtQuery = new TextField();

 public static void main(String args[]){

 public void start(Stage stage){
  VBox root = new VBox(10);
  root.getChildren().addAll(txtQuery, chart, table);
  stage.setScene(new Scene(root));
  txtQuery.setOnAction(e -> {
   List textSentiments = getTextSentiments(txtQuery.getText());
   chart.setTitle("Sentiments for \"" + txtQuery.getText() +"\"");
  txtQuery.setPromptText("Enter your query for text and press enter");

 private void initializeApp(){
 private void addTableColumns(){
  TableColumn textCol = new TableColumn<>("Text");
  textCol.setCellValueFactory(new PropertyValueFactory("text"));
  TableColumn polarityCol = new TableColumn<>("Polarity");
  polarityCol.setCellValueFactory(new PropertyValueFactory("polarity"));
  table.getColumns().setAll(textCol, polarityCol);

 public void fillTable(List textSentiments){

 public void fillChart(List textSentiments){
  long polarity0Count = textSentiments.stream().filter(s -> s.getPolarity() == 0).count();
  long polarity2Count = textSentiments.stream().filter(s -> s.getPolarity() == 2).count();
  long polarity4Count = textSentiments.stream().filter(s -> s.getPolarity() == 4).count();
    new PieChart.Data(":(", polarity0Count),
    new PieChart.Data(":|", polarity2Count),
    new PieChart.Data(":)", polarity4Count)
 public List getTextSentiments(String q){
  return textSentimentService.getSentiments(query(q)).getTextSentiments();

 public List query(String q){
  if(q == null){
   return Arrays.asList("obama is awesome", "obama sucks", "obama eats potato");
   // TODO: add new providers and adapt the app to support it
   return  searchService.search(q, SearchService.Provider.TWITTER);

Notice we use Java 8 features to deal with the List of  TextSentiment objects! Also, we used Lambda in some places of the code, can you spot it?

Do you wanna play with it?

The app code is on github and it uses Maven. if you want to play with it, use git app to clone the repository locally and build it. Remember to use Java 8 and Maven 3.x!  Steps:

 Clone the APP

$ git clone https://github.com/jesuino/sentiments-app.git
Go to the app root directory  and use the following command to build:

$ cd sentiments-app
$ mvn clean package

You can also run a test:

$ mvn test

Finally, you can run the App using the run.sh script. Remember to set JAVA_HOME correctly! run.sh content:

export JAVA_HOME=/opt/java/jdk1.8.0/
mvn package exec:java  -Dexec.mainClass="org.jugvale.sentiments.view.App" -DskipTests

App screenshots

Searching for happy words:

Searching for sad words:


 That's all, folks. Just wanted to share this small fun app with you. I made the code really simple because I'm lazy to someone extend it. If you extend it, please, send me pull requests :)
  • Extending by adding other text sentiment analyzer. Lots of text sentiments APIs here.
  • Extending by adding other source then twitter, for example:  news sites, google plus, facebook etc.
  • Improve the view by adding more controls, a "waiting screen", etc
  • Add a CSS to improve the look of the app

Nenhum comentário:

Postar um comentário