Our previous study extracts human readable topics given a set of microblog posts. Based on the idea of identifying the topics of a crowd of microblog users, we have recently came up with semantically representing microblog topics for machine consumption. Source code of the prototype is published. To install topic identification approach in a linux machine follow the following steps.

• Install R
• Make sure that Rscript is running
• Install php-cli (Php command line interface) version>5
• Make sure that php-curl is installed
• Make sure that shell_exec is working in PHP-cli
• Obtain a TagMe API key
• Download the SBounTI package and extract it in an empty directory
• Edit cfg/config.php according to need (such as base urls of resources that will be produced and the TagMe API key)
• Obtain a microblog post dataset about 5 thousand posts, either
  • in a file format of short texts in each line
  • or in a raw file retrieved from Twitter streaming API
• Issue command:
  • ./sbounti <filename> "<dataset_name>" "<start_date>" "<end_date>"
    for the text file
  • ./sbounti <filename> "<dataset_name>"
    for the raw Twitter streaming API file
  Where <filename> is the file name of the file that has short messages, <dataset_name> that is used in the explanations of the resources expressed in OWL, <start_date> and <end_date> are valid start and end date-times of the post set in the format as in example: Wed Sep 21 11:01:56 +0300 2016.
• The produced OWL file contents are written to STDOUT. So, you may want to redirect the output to a file using "> filename.owl" at the end of the command.
• If you have questions please contact Ahmet Yildirim

I am doing experiments with tweets. For each tweet or each set of tweets, several APIs need to be called trough http protocol. This takes time. I often repeat the same experiment, which means issuing the same http request over and over again. Each request takes at least half a second. Considering a million tweets, one experiment takes half million seconds which makes over 5 days. And, this is the case for only one request per tweet.

In order to overcome this, I cache requests and responses. I made a library for php. Below I will be giving the codes. The library issues curl. But before issuing curl, if cache option is set, it first check if there is a cached content for this request. If there is, it simply returns the cache content. Otherwise, it requests the content over the network, and caches it for further requests.

Caching is done by saving the response to a file named by the md5 of the request url. Since the number of files in the cache directory is too high, I applied a two level strong mechanism which gets first two chracters of the md5 hash and saves the file in the directory named with those two chracters. This reduces the number of files in one directory, enabling a two level look-up in the file system.

Below is the simple code. It, for sure, needs further improvement but, for now, it works for me.


function curl_get($url, $cache=false)
{
$md5filename=getFileName("urlcache",$url);

if($cache==true)

        {

        if(file_exists($md5filename))

                return file_get_contents($md5filename);

        }

$defaults = array();

@$defaults[CURLOPT_URL] = $url;

@$defaults[CURLOPT_HEADER] = 0;

@$defaults[CURLOPT_RETURNTRANSFER] = TRUE;

@$defaults[CURLOPT_TIMEOUT] = 0;

$ch = curl_init();

curl_setopt_array($ch, $defaults);

if( ! $result = curl_exec($ch))

        {

        $retry = 0;

        while($retry < 10){

                sleep(10);

            $result = curl_exec($ch);

        if($result)break;

        if(!$result)

                trigger_error(curl_error($ch));

        }

curl_close($ch);

if($cache==true)

        {

        $f=fopen($md5filename,"w");

        fwrite($f,$result);

        fclose($f);

        }

return $result;

}

function getFileName($infix,$data)

{

$md=md5($data);

$st=substr($md,0,2);

$md=$st."/".$md;

@mkdir(__DIR__."/../caches/$infix/$st");

return __DIR__."/../caches/$infix/".$md;

}

Last year, we have published a very exciting research article. In this article, we have discussed if Wikipedia page titles can be used to represent topics that are talked about in Twitter and proposed an approach to do that.

In contrast to existing topic extraction methods that extracts topics from only one tweet, our approach extracts topics from multiple posts. We assume that, elements of topics that crowds talk about distribute to multiple tweets by multiple users.

Our approach also differs in representation of topics in contrast to approaches that represent topics as a set of words such as LDA, phrases, or representative tweets. Our approach represents topics using Wikipedia page titles.

We used a simple computation. We compared contents of the tweets with contents of the Wikipedia pages using cosine similarity. This computation is not easy as Wikipedia has over four million articles. You need distinguishness of words in tweets. To measure distinguishness of words, inverse document frequency (idf) of words have to be computed.

For the details on computation you can g see the published paper. But now I want to show interesting results here.

We have experimented our method on tweet sets gathered during 2012 US elections debates. Debate tweets are interesting because they span a  wide range of topics such as women issues, tax, unemployment, social services.

The following figure shows two scores of two topics over debate times. The first plot (a) is for the Wikipedia page "Big Bird", and the second plot is for "Christianity and Abortion".

Big bird received high scores around 28th minute of the first presidential debate. Mitt Romney (the former Republican party candidate for elections) said something like : I will cut subsidy to PBS even if I love big bird etc. It received a quick response from the twitter environment.

Abortion is a critical subject in the United States. Whoever the candidate is, he needs to say something about abortion. The moderator asked both vice presidential candidates about their opinion of abortion as they are Catholics. Since public is sensitive to this matter, it also received a quick response. It should be noticed that no one used "Christianity and abortion" phrase in tweets. But our approach revealed this topic by considering aggregation of what is talked about.

The following picture is a heat map of several topics over debate times.

Third debate was about foreign issues. Our approach gave high scores to related topics such as Foreign relations of china, Iran-United States relations, Israel-United States relations, and Foreign policy of the United States. It can be tracked from the darkness of the topics from the heat map that, in the second half of the third debate, in order, Israel-United states, Iran United States, Israel-United states again, Osama bin Laden, and China are talked about.

Other topics are also meaningful. For instance, obamacare (patient protection and affordable care act) was mostly the issue of the first presidential debate. This can be seen from the heat map.

We have also experimented the 2016 debates between Donald Trump and Hillary Clinton. The results will be published soon.

Stay tuned!! :)