Downloading Twits using Twitter Public Stream API

In this post will try to show how to connect to Twitter Public Stream API and download live Twits. We can save these Twits in any format like comma or pipe delimited. These Twits can be use for Data Mining or Data Analysis. Have used this Twits to analyse the current News in given region.

This program creates files of 50MB each, and keep rotating till the time the stream is running. we can run Map-Reduce program using Hadoop Framework to analysis the data at much faster rate.

Prerequisite:

1. Need to have Twitter Application
2. Twitter OAuth Tokens
     https://dev.twitter.com/docs/auth/obtaining-access-tokens
3. Twitter4j Libraries
    http://twitter4j.org/en/


Below are the twitter4j API used:

TwitterStreamFactory : This API returns the object of type TwitterStream

We need to set below authorization tokens to connect to Twitter Public Stream. You will find below tokens once you create Twitter Application.

-Dtwitter4j.oauth.consumerKey=XX  
-Dtwitter4j.oauth.consumerSecret=XX
-Dtwitter4j.oauth.accessToken=XX
-Dtwitter4j.oauth.accessTokenSecret=XX

If you are running this program inside Firewall you need to set proxy details to connect to Internet.

-Dhttp.proxyHost=proxy 
-Dhttp.proxyPort=80 

Have created simple TwitterListener.java class which implements StatusListener Interface. This class is act as the Listener to public twits. It returns the Status object which basically is the Java object the Json string returned by the Twitter. This object contains all the information related to that Twit eg. Location, Message, DateTime etc.

Have created simple Logger (TwitterLogger) to log the Twits in file. Have used "|||" as a field separator and new line as record separator. The logger will rotate the file if the size reach around 50MB.

Also, you will find the window's command line scripts to start the Twitter downloading. It basically calls (TwitterConnector) main method. Set user.home variable where you want the Twits to be download. The user.home must have logs folder.

In one day it downloads around 1-2 GB data. We can use this data for analyzing information or run few BigData programs.

References:

https://dev.twitter.com/docs/streaming-apis
https://dev.twitter.com/docs/auth/obtaining-access-tokens
https://dev.twitter.com/docs
http://twitter4j.org/en/javadoc.html

Code:
TwitterConnector Download











Popular posts from this blog

JAVA embedding in Oracle SOA 12c

Image
In this post we will explore how to EMBED Java or call Java API from Oracle SOA BPEL process. In order to demonstrated this, I have created a simple JAVA project with 3 classes. 1.        Account.java -> This class represent Account DTO       2.       Party.java -> This class represent a party (customer) DTO       3.       GetNumber.java -> This class contains the method that takes Party DTO as input and apply some business logic and return party’s AccountDTO in response. public Account getNumber(Party party) The above classes are created to simulate the calling of actual business logic API’s. Account.java Party.java GetNumber.java public class Account {     public Account() {         super();     }     String account;     public void setAccount(String account) {         this.account = account;     }     public String getAccount() {         return account;     } } public class Party {     public Part
Read more

Passing/Receiving HTTP header in SOA BPEL

Image
In this example we will explore how to pass or receive the parameters from HTTP header. We will explore below two scenarios, To receive the HTTP header from SOAP request. To pass the HTTP header while calling soap service. SOA version used: 12.1.3c To receive the HTTP header from SOAP request: First we need to add the property “oracle.webservices.http.headers” under binding.ws in Service element in composite.xml. This property contains the comma separated parameters that we are expecting in request. Example,  ENV,VERSION. <binding.ws port="http://xmlns.oracle.com/SOA_Learning/HTTPHeader/HTTPHeaderTest#wsdl.endpoint(httpheadertest_client_ep/HTTPHeaderTest_pt)">          <property name="oracle.webservices.http.headers">ENV,VERSION</property> </binding.ws> Now in BPEL source under receive activity, we need add <fromProperties> element which will copy the parameters from HTTP header to BPEL variable.
Read more

Integrating Weblogic with IBM MQ over JMS

Image
We can integrate weblogic with IBM MQ over JMS by creating Foreign JMS server with Links in weblogic server. We need to create .binding file which will acts as file based JNDI provider to create JMS binding to MQ resources. Will first explain how to create .binding file and then will configure weblogic to use the same. Steps to create .binding file: Firstly we need to add below jar’s in CLASSPATH jms.jar com.ibm.mq.jar com.ibm.mqjms.jar jta.jar connector.jar jndi.jar providerutil.jar fscontext.jar Create JMSProperties.config file with below properties INITIAL_CONTEXT_FACTORY=com.sun.jndi.fscontext.RefFSContextFactory PROVIDER_URL=file:/tmp/mqm/jndi SECURITY_AUTHENTICATION=none Run below command ./JMSAdmin -v JMSProperties . config InitCtx> def qcf(ForeignCF) TRANSPORT(CLIENT) HOST(localhost) PORT(1414) CHANNEL(MQCHANNEL) QMGR(MQMRG) def q(MQ1) qmgr(MQMRG) queue(MQ1) def q(MQR1) qmgr(MQMRG) queue(MQR1) display ctx
Read more