Twitter API: Searching tweets, replies

Get started with Twitter's API, searching tweets, loading replies

This project is on Github repo as an API server. Clone this to use twitter API immediately(after you've created your twitter developer app), to search for tweets, load replies to tweets.

What you need to get started?

You will need a twitter developer app to access the Twitter API. The App will provide you with two keys: API key, and API secret which we will use for requesting data from the API endpoints. Follow the following steps to get yourself the necessary keys.

1. Apply for a twitter developer account
2. Create a twitter developer app
3. Generate your app's keys on the app's details page under Keys and tokens tab
4. Open terminal and run the following command to generate your app's Bearer token. curl -u '<API key>:<API secret key>' --data 'grant_type=client_credentials' 'https://api.twitter.com/oauth2/token'

Replace <API key> and <API secret key> with your app's key and secret.

Note: With Bearer token, your app can only search for public data. It can not post tweet, read/write messages from your account. For such purposes, you'll need the Access token and secret too. You can generate them on the same page where you generated your keys. For our use case this will be sufficient.

Project setup

Nodejs Typescript boilerplate here.

You are now ready to use Twitter API. We are using a Node.js application and writing in Typescript, but you can use any language you prefer. But depending on your language of choice, you may find a twitter library or may have to build your own. For Node.js, we have a library called twitter, which we will use to make the requests

Add the dependencies

1. Twitter library npm install twitter
2. Types npm install @types/twitter

Modify environment variables

Open up the src/.env file and add your app's keys and token.

TWITTER_APP_API_KEY=<your-api-key>
TWITTER_APP_API_SECRET=<your-api-secret>
TWITTER_APP_BEARER_TOKEN=<your-bearer-token>

Add this to the config file src/config.js to load these env variables

export const twitter = {
    consumer_key: process.env.TWITTER_APP_API_KEY,
    consumer_secret: process.env.TWITTER_APP_API_SECRET,
    bearer_token: process.env.TWITTER_APP_BEARER_TOKEN
}

Let's search for some tweets. Shall we?

In src/app.ts file write the following code.

import Twitter from 'twitter';
import {twitter} from './config';

let tw = new Twitter(twitter);

tw.get('search/tweets', {
    q: '#webdevelopment',
    count: 2
})
    .then(res => {
        console.log('Response: ',res);
    })
    .catch(err => {
        console.error(err);
    })

Run this code by npm run start and you should see Twitter respond you with two tweets as an array, and some metadata about the search itself

Search parameters

Here, we are only using two properties q and count. we are searching for tweets that contain the hashtag #webdevelopment, and we are retrieving only 2 tweets.
See the full list of properties available here.

Search operators

For standard search(free with 7 days of data), your search query q can use a list of standard search operators. In our example we are searching by hashtag #webdevelopment.

Pagination

For obvious reasons, Twitter allow only a maximum of 100 tweets to load at a time. You can load the next tweets(older), or refresh the search(newer tweets) using two parameters max_id and since_id. When you get response from twitter, you get one search_metadata field which contains next_results and refresh_url. From these two query strings, you can filter out the max_id and since_id respectively. Then if you want to load the older tweets simply add the max_id to your parameters.(alongside q).
In my repo, there is a PaginationController class which automatically filters out the max_id and since_id if you just supply next_results and refresh_url respectively.

Schema for tweets

While twitter gives a whole lot of detail with tweets, we do not need all that. For example tweets contain both id_str and id, but we will only use id_str as id as it contains the original value. Also what's good is typescript, if we don't use a type for tweet. So, I have created a schema for handling tweets, and also a filter function to filter out tweets in our schema from twitter responses. Check them out at my repo. They are in file src/api/twitter/schema.ts and src/api/twitter/filter/tweetFilter.ts

Loading replies

Loading replies is a bit tricky. Because, twitter does not provide us API to load replies for a tweet directly. That's sad. But thankfully, twitter standard search operators can be used to fulfill our need. It provides us a to:screen_name operator which allows us to load replies to a twitter user. The screen name is the twitter handle like @thisisrituraj, without the @ sign. So to search for replies to my account, I would search using q: "to:thisisrituraj".

This will load replies to my account. But I wanted replies for a particular tweet of mine. Not all. That's what we have to filter out by looking a field in the responded tweets. The in_reply_to_status_id_str field which holds the id of the tweet for which this tweet is a reply.

We have to load all the replies to my account, using pagination, and then filter out replies to a particular tweet using its id. But do we have to load all replies to filter out replies to just one tweet. Unfortunately, twitter does not provide us with a tweet's reply count to know how many replies are there for a tweet. BUT..., we it provides us with the timestamp when the tweet was created(created_at field inside a tweet). And we know that, replies to a tweet can not predate the tweet itself.

So, the steps to load replies are

1. Get the id of the tweet(id_str) for which we wan't to load replies to.
2. Get the user's screen name who posted this tweet.
3. Load replies to this user using to operator
4. Filter out replies by comparing in_reply_to_status_id_str with id_str
5. Use pagination to keep on loading replies until one of the following occurs
   • We have run out of replies. This means no more replies are there, or we have loaded replies for past 7 days
   • We find a reply whose timestamp predates the timestamp of the tweet

Here's the code for loading replies. (This is an excerpt from my repo)

export default class TwitterClient {
    // code omitted for brevity

    private async loadRepliesForTweet(tweet: ITweet): Promise<ITweetsResponse> {
        let { user, id_str } = tweet;

        // save tweet's post datestamp
        let tweet_at = new Date(tweet.created_at);

        // load all replies to this user
        let allRepliesToTweet: ITweet[] = [];
        let replies: ITweetsResponse;
        let max_id = null;

        do {
            // load all replies to user
            replies = await this.searchTweets({
                q: `to:${user.screen_name}`,
                max_id,
                include_entities: false,
                count: 100
            });

            // select only replies done to THIS tweet
            replies.tweets.forEach(tweet => {
                if (tweet.in_reply_to.status_id_str === id_str) {
                    allRepliesToTweet.push(tweet);
                }
            })

            // Get max_id from next_results
            if (replies.search_metadata.next_results) {
                let next = <string>replies.search_metadata.next_results;

                // Use PaginationController to get max_id
                max_id = PaginationController.getParameterFromQueryString(next, 'max_id');
            }
            else {
                // BREAK loop if no more results exist
                break;
            }

            // get last reply tweet's post datestamp
            let last_reply_at = new Date(replies.tweets[replies.tweets.length - 1].created_at);

            // BREAK loop if last reply is earlier than tweet itself
            if (last_reply_at.valueOf() < tweet_at.valueOf()) {
                break;
            }

        } while (true);

        return { tweets: allRepliesToTweet, search_metadata: replies.search_metadata };
    }
};

That's all folks!

That was all about using the Twitter API for searching tweets and loading their replies. We saw, that although Twitter does not allow us to load replies to a tweet directly, we can use its Standard Search Operators and some coding to solve that.

How do you like this post. Do you have any idea about building something with Twitter API. Let me know. I develop software applications and am very much into problem solving. Connect with me on linkedin