fetch

Twitter API with Python: Part 1 -- Streaming Live Tweets



Sharing buttons:

okay so in this video in the series of

videos we'll be making use of the twe P

API and this is an API that we can

access via Python which will allow us to

very easily stream tweets in real time

directly from Twitter so in order to

follow this series of tutorials what

we're going to be doing is installing

tweet P which is a Python package so I'm

going to assume that you have Python

installed on your machine and if so all

we'll need is the twe P module so let me

just give a brief overview of what we'll

cover in at least this video this video

we're just going to create a Python

sorry well we are going to create some

Python we're going to first create a

Twitter application and in order to do

that you will also require a Twitter

account so if you don't have a Twitter

account go ahead and create one it's

pretty easy to do doesn't need to be a

main Twitter account or anything like

that you just need to sign up for

Twitter in whatever way you wish to do

so

once you do that you will have the

ability to create a Twitter application

and then well write Python code that

will interface with that application to

stream tweets in real time based on

keywords or hashtags now what we'll do

in this video is we'll just write a very

simple application to accomplish just

that so we'll see tweets given a set of

keywords and then in later subsequent

videos we'll see what we can do to

analyze the content of the tweets that

we get back maybe we can produce some

nice graphs based on that data and then

some other fun things that we can do

with that data like possibly sentiment

analysis well which will most likely be

part of the natural language processing

series of videos so keep an eye out for

that as well so I guess one thing that I

want to mention as well is that all of

the code as usual for all of these

things will be provided on the github

page and all the links that I'm talking

about here all the links that I have

open on this browser are going to be

available below in the description so no

need to write them down but just refer

to that and you can see the code that we

use in this in all videos another thing

I want to mention too is that if you

have any problems or questions about

twee pea which is the primary

that we will be using in this tutorial

series the documentation is doc stats

weaboo org again link will be in the

description and more further information

about this module is there and it's very

descriptive and well-written so I will

refer you to that so before we get into

writing any Python code as I mentioned

we need to actually go ahead and create

a Twitter application so in order to do

that what we need to do is navigate over

to apps Twitter comm again this will

require that you do have a Twitter

account and are logged in if so you

should be presented with a dashboard

that looks probably similar to this

depending on what time when you're

watching this video you can see that I

already have an application here listen

to under my Twitter apps that was used

for something else I was working on just

for fun and what were your going to want

to do you might not see anything here if

you don't have any apps in this

application management section you can

click on this create new app button

that will take you to a form which just

asks for some really simple details

about what this Twitter application is

going to be doing so I've already kind

of did a dry run of this and I have some

information for what we'll be filling in

so you can put in whatever name you like

I'm going to put in this name this is

just the name of the Twitter application

lucid programming Twitter app a

description which is also required so

I'm just going to say a test Twitter

application for a YouTube tutorial and

then a website so I'm just going to put

in my own homepage here just for good

measure I suppose callback URL this is

not particularly necessary so we'll just

leave this blank make sure you click

this developer agreement and then you

can click on create your Twitter

application so if you do so then you are

presented hopefully with this page that

says your application has been created

and all of the relevant information for

your application will be stored here in

terms of how to access

I guess credentials and things like that

so what you're going to want to do is

actually click on the Settings tab right

here so if you do so you will see a

you'll see four things listed consumer

key consumer secret access token and

access secret I'm not going to click on

that because the

I guess the keys are unique to each

program and I guess showing those things

is I think it's against well it might be

in poor taste I don't exactly know if

there's any like security breach if I

share those things but I guess just for

good measure I'm just going to let you

click on let yourself look at what those

are keep them secret and then we're

going to use those to interface with our

application I guess for one thing if you

do have a Twitter application that's

interfacing with Twitter you wouldn't

want somebody to get access to your

tokens because then they can manipulate

it and they could you know possibly

abuse it and you know that wouldn't look

good for your Twitter account so try not

to let that happen

so we're going to use those four

credentials to authenticate the Python

program that we will now be writing so

with that said minimize this and let's

get to writing some code so the first

thing I'm going to do is I'm going to

create a credentials file which is going

to store those four things which you

should see on your settings tab so I'm

going to create this as Twitter

credentials dot PI so what we're going

to do here is we're just going to create

some variables which will be accessed in

our primary program so I'll just put in

a comment variables that contain the

user credentials to access Twitter API

so the first one is access token the

next is access token secret there's also

the consumer let me spell it properly

consumer key and the consumer secrets so

what I'm going to do is I'm just going

to whoops I'm going to define each of

these is just Python strings so I'm just

going to assume that you can go over to

the Settings tab back to here click on

the Settings tab and then copy and paste

in between these quotes the access token

access token secret consumer key and

consumer secret save this file and make

sure that this file is saved in the same

directory that you'll be using all of

your that you'll be writing all of your

Twitter API access Python code in

I'm just going to save this right now

and that's that so if I look here I have

Twitter credentials pi in this directory

in the same directory I'm going to

create another Python file which will be

the brains of what we'll be doing which

as I mentioned before will just be in

this video at least streaming the tweets

directly from Twitter in a real-time so

let's create a file which we'll call twe

P streamer dot PI so before we get into

writing this one thing that I probably

should have mentioned earlier that I did

is that

oops that was not supposed to go there

let's see let's try that again there we

go you need the tweet be module of

course so as I mentioned I'm going to

assume that you have Python installed

I won't assume that you have tweet be

installed so if you don't have it

installed what you'll need to do is

access it through pip pip install twee

pea if you run this command into your

terminal it will go ahead and install

for me I already have it installed so it

says requirement already satisfied so

sorry about that I've forgot to mention

that earlier so back to the code so back

to here let's go ahead and start

importing some of the things that we'll

be making use of that are coming from

the twe P library so the first thing

that will import is tweet be streaming

import stream listener this is a class

that's from the Tweety module that will

allow us to listen to the tweets kind of

the firehose of tweets as they as they

come based on certain keywords or

hashtags we'll also need to import

another thing for authentication

so we'll say from TP imports ooofff

handler this class is going to be

responsible for authenticating based on

the credentials that we store it in the

other file for associated with the

Twitter app and then we're going to have

one more thing from tweet P imports

stream and I guess another important

that we require not from twee P is the

credentials file that we created so

imports Twitter credentials

and again this should be located in the

same folder as this file that we're

ready to currently so the first thing

that I want to do is I just want to

create a class which will allow us to

print the tweets so a very simple thing

and then we'll go back over it and kind

of make it a little bit more robust and

we'll refine what it's doing but simple

first so let's create a class and we'll

call this standard out listener and this

class is going to actually inherit from

stream listener so it's going to inherit

from the stream listener class and the

stream listener class provides methods

that we can directly override so one of

the methods is called on underscore data

class method that takes in a parameter

data and there's another one that we can

also override called an error which also

is cost method and takes a status

variable so what are these functions do

what are they responsible for on data is

an overrun method which will take in the

data that is streamed in from the stream

listener so the one that is listening

for tweets and then it's going to print

we can do whatever we want with that

data that tweet so what we're gonna do

just to make it very simple is we're

going to print out the data that we get

and then we're just going to return true

to make sure that everything went well

we will go back and make that a bit more

robust because that's a very simplistic

at the moment odd error this is what a

method that we are overriding from the

stream listener class that happens if

there's an error that occurs and one

thing we can do you again which is just

very simple is we can just print out the

error which is passed in through this

status variable we're just printing it

on on screen so if we encounter an error

what will happen is this method will be

triggered and will print the status

message of that error to the screen so

the next thing that we're going to want

to do is just actually create a method

sorry an object from this standard out

listener class that we just created and

then actually get on to streaming the

tweets so this will be in the main part

of the program so we'll say if name is

equal to

made and what will first do is create a

listener object so I'm gonna say

listener is equal to standard out

listener like that so again this is just

an object of the class that we just

created which is inheriting from the

stream listener class and then what

we're going to do is we're going to want

to authenticate so we're gonna want to

authenticate using the credentials that

we had stored in the other file so we're

gonna create this variable called auth

and we're going to say Oh auth handler

which is the class that we're importing

from tweety which is going to be

responsible for actually authenticating

to our code and we're going to pass it

in the credentials so we're gonna say

Twitter credentials dot consumer key and

Twitter credentials dot consumer secret

so this in order for us to define this

auth object of the off handler class it

takes these two arguments that we need

to pass in and then in order for us to

complete the authentication processes

we're going to say off that set access

token and this is a method which is

provided from the OAuth handler class

and what this takes this method also

takes two arguments and that takes the

access token and the access token secret

at this point our application hopefully

should be properly authenticated so

we'll create a Twitter stream based on

the so we'll call this will create a

variable called stream equal to this

lane which is the class stream that we

imported above and we're gonna pass the

two things the authentication token to

verify that we've actually authenticated

it properly and then the listener object

that we created and the listener object

is just responsible for how do I deal

with the data the tweets and how do I

deal with the error if I encounter an

error one final thing that we can do is

we can filter the tweets because

otherwise if we just run this stream if

we run this listener it's just going to

stream a ton of tweets at us which might

be some of them might be interesting

some of them might not be let's say we

want to stream tweets that are focused

on some keywords or hash tags or

something like that

so what we can do is we could say

streamed

filter this is a method that is also

provided by the stream class and that

what this takes is a list one of the

things that can take is an optional

parameter of a list which is called

track and in this track list we can

provide it a list of things which if the

Twitter sorry if the tweet contains any

of these lists objects then it will

apply it and it will say I'll add this

to the stream so let's see let's just

add Donald Trump

let's Hillary Clinton and Barack Obama

and let's say Bernie Sanders so just

some politicians in there so we'll

filter tweets based on this list of

keywords so we can go ahead and write

this to a file and then we can go ahead

and try to see what happens when we run

it so python tweet be streamer now if we

run this we get so let's see a 401 let's

check why that happens to be I think the

reason for that was because I didn't

fill in any of the consumer key consumer

secret or anything like that I just

written it from what you saw before all

of those things were completely blank so

don't be me don't be an idiot go ahead

and make sure you fill that stuff in and

once you do if you do that hopefully

that should run and things should start

coming down the screen there so like we

see a ton of look at seemingly garbage

just showing up on the screen here these

are tweets specifically these are let me

just kind of stop this here so if you

don't stop it it'll just keep you know

just keep streaming so this whole blob

here this is one tweet it's not the

person didn't tweet this thing out of

course what this is is it's a JSON

formatted dictionary object and each of

the fields in this braced thing here

contains information about the tweet so

this says the tweet was created today

this is today this is the you know ID

that Twitter associates with this

particular tweet apparently it was a

retweet by this person here here is the

actual text so Donald Trump that's one

of the keywords that we had in the

in the list check so 2019 state of the

unit speech is January thirtieth

TV ratings so yeah there's just some

stuff that he's saying there what what

other content do we have on this thing

that's interesting let's see so it tells

you also about like who retweeted it how

many times been retweeted there's a

whole ton of information in this little

thing here I think even the platform

coordinates possibly of where this was

tweeted from in some cases sometimes

that's not provided but there's a ton of

information in this thing so there's one

suite here and what we're gonna be doing

it's subsequent videos is actually going

through each of these and seeing if we

can extract anything interesting any

insights from these things and just kind

of have some fun with it so back to the

code so what we have at this point is

just something that prints out tweets to

the screen which is totally fine that

it's proof of concept we've got tweets

coming in but I'm just going to clean

some of this up to make this a bit more

robust so that in subsequent videos

things are just kind of a little bit

more

well robust I guess is the right word so

let's go and create another class which

we'll call let's call this Twitter

streamer so this is going to be a class

that will define for ourselves which

will be responsible for actually

streaming the tweets just makes it a

little bit more concise so we'll call

this stream tweets and this is a class

method so it takes the object of self

I'm kind of thinking ahead here and

assuming that instead of just printing

tweets to the terminal like we saw there

perhaps I want to save tweets to a text

file or a JSON file so that way I can

process them for whatever purpose I see

fit later so I'm going to allow the

ability for us to pass in a name which

will be the file name of the tweets that

will pass in so let's say fetched tweets

file name I'll call that so again that's

just the file name of where we want to

let's say write our tweets to instead of

showing them on the terminal we can do

both but we don't have to and then I'll

also have a hashtag list here so this

will just be the list of keywords or

hashtags we wish

you filter the tweets out by so we're

really just kind of reworking some of

the code that we have that's the proof

of concept into something that's a

little bit more modular so let's go

ahead and add a comment just to make

this look a little nicer this handles

Twitter authentication and the

connection to the Twitter streaming API

so we're going to pretty much just take

this code that we wrote down here and I

think we're just gonna well let's just

copy that even remove it move that over

here let's go ahead and indent that so

we'll create the listener object that's

all good

authenticate do that so instead of

hard-coding this here I guess what I'm

going to do is I'm going to remove this

and I'm going to put in the hashtag list

that we passed into the function so a

bit more modular so we can just create a

Twitter stream or object instead of

doing that whole business that we had

before in the main part of the code

another thing that I guess I want to do

in this standard out listener class that

we defined is well let's see let's just

let me just add a few more comments here

so I want to add a comment for this

class this class is a class for

streaming and let's say processing live

tweets that looks good and then here

let's do another one this is really this

is a basic listener class that just

prints at this moment anyway just prints

received tweets to standard out so that

might change over time as we made this

more complicated so I guess the first

thing that we'll do is we'll create a

constructor for this thing because what

I want to do is I want to create a state

about let's enter a class or object

rather and that may be associated with a

filename that these are going to be

writing to so where do we want to store

the tweets so I'm gonna say fetched

tweets filename and this is a class

variable so I'm going to say self dot

fetch tweet

file name is equal to fetched tweets

file name so this odd data method is a

little bit it could use a little bit of

something to make it a little bit better

at dealing with possible errors so I'm

going to have a try except statement

here so I'm going to say try let's say

print data and then actually I'm not

going to put anything well okay let's

say it let's say that we also want to

write this stuff to a file too right so

we've already created this file name

that we've defined the constructor of

this class let's say with open self dot

fetched tweets file name and will append

because we keep once we want to

continually add the tweets as we stream

them from the API stf little writes the

data so TF that write data so if you

don't want to print it out and also

write it you can get rid of that print

statement but it's totally up to you if

that what okay will return true and then

we'll have an accept case here so accept

will say base exception and then if that

exception is hit we'll say print error

on on data just to make sure that we

know if the method that we are in and

then let's go ahead and print out let's

print out the actual error message so

we'll print out the string of E and then

we'll return true okay so I think that's

okay the error on error message I think

is alright for now and then okay so now

we had to change a little bit here in

this main function so let's go ahead and

use our new class to do pretty much what

we did before but just make it a little

bit more nice and modular so what we'll

do is we'll create a hashtag list here

well I guess create it as let's say

Donald Trump Hillary Clinton

Barack Obama and Bernie Sanders and

let's see so we could also say let's

define our affection tweets filename as

let's just call it tweets JSON so again

as we saw from the output of the

terminal those tweets the braced objects

are they can be formatted as JSON it's a

little bit easier for us to deal with

that format if we want to read little

later you could put that txt there it's

totally fine but I'm just changing the

extension to JSON it's not necessary if

you don't want to do that okay so now

let's define a Twitter streamer object

so I'll say Twitter streamer is equal to

Twitter streamer and then we'll say

Twitter streamer there it is dot stream

tweets which is the method that we

created up above and we'll send it takes

two things the file name that we want to

write to and then also the list of

keywords that we're looking for so that

just made that a little bit more concise

and everything a little bit more modular

and that will kind of get us started on

this whole stream of tweets lien so in

subsequent videos we'll go ahead and

look at precisely what data do we get

from these tweets and we'll see if we

can do some interesting analysis on

these tweets as well so thanks again for

watching stay tuned please let me know

if you have any feedback comments

subscriptions are appreciated comments

are appreciated so any of those things

please let me know and thanks again for

watching

have a great day bye