fetch

How to Fetch data from any website using ESP8266 | IoT Projects



Sharing buttons:

hello youtubers welcome to my next

tutorial on how to fetch data from any

website you want so this is the Sunday

morning seven Feb 2016 in India time is

10:30 5 a.m. and it's the winter morning

so let's see you what's the temperature

in my city so I am press the temperature

button so it is processing some data and

the temperature here is around 22 degree

Celsius blue it's not that much cold so

yeah so this is all we got to make today

we will learn how to fetch data from any

website so let's get started so

basically I have used this touchscreen

module so that it kept easy for you to

understand this project but you can use

anything you can use any LCD or

7-segment display to display the integer

numbers anything you want yes you

required is a Wi-Fi module okay so first

of all first of all let's declare the

less discuss the theory behind this or

how this process is working then we will

come to the code so basically let's take

example of live time in India

okay so life time India okay so I will

go to one of the website okay let us say

time and date calm okay so this is the

website and this is the current time of

India let's go this HTML page inspect

so basically when we request this link

or this URL it gives it in response this

much or this code in response we get

this code now from whole this or this

long code we want only the data of the

current temperature so we want something

something to request or something which

we request and in response we get only

the time only the required data so for

example your time is again inspect

where's the time time time time where is

the time I'm going to select this and

again inspect so this division okay so

here is the time now we want to target

this particular data so for that we

require an API now API stands for the

application program interface in detail

if you want to study about API you can

watch another videos but for those who

want overview of what an API does for

that I will explain API is basically

interface through by using API we can

target the required data from whole HTML

page and we can fetch or we can get the

data required data only from the whole

page available now the question is how

to make an API now API can be made

manually and for those who don't know

about coding like me can go to this

great website called things big calm

things be calm is great grid website for

starting into getting into the IOT

Internet of Things it's basically a

framework so it will automatically

provide us the required API any from any

website offer on any server so let's see

what's the procedure of making an API so

first of all you need to make an account

I already logged into my account so we

will go to apps apps and we will go to

the last block things HTTP or thing as

TTP simplify device communication with

web services and API is so here this

thing we will require so we will make a

new thing HTTP now the now we need to

provide some of the information that

from which website we want and for which

data from the whole page we want we need

to provide some basic data only and it

will automatically generate an API so

this one we will name that API suppose

five time India okay so now we will do

we have to provide the URL so we will

provide the URL of the website from

which we need to fetch the data so copy

and paste the URL so this is the sec

step authentication username password

leave it blank and we need to get SST

HTTP is all about getting and posting so

here we need to get the data so we'll

select the get and the version of HTTP

is 1.0 1.1 and the last and most

important front waters thing is parsed

train now what is that we need to

provide a tag or what you can say huh a

tag or an identity of the data which we

want to fetch so here is the time so we

will right click the time and copy XPath

XPath will copy the tag like something

or ID yes it will fetch the ID of that

data from that page so we will provide

this ID to pass string okay so this was

a pound of data we need to give and it

will automatically automatically

generate an API key this is an API key

particularly for this for fetching the

lifetime anyone having this key can face

that lifetime from that website our time

indeed.com

so we will save this thing as TDP let's

see the miracle of API so this is the

API and this is a get request we need to

simply post this link on to our browser

and let's see what's happened okay

so rather than the whole HTML page we

caught the required data that is the

time of lifetime in India that is 10

41:38 a.m. so this is the base of this

project making an API indent becomes a

very simple most simplest using this

thing's pick calm I'm sorry I am NOT

going into detail about API because the

tutorial will be

more longer so you can watch the API

what what we'll learn about API on

anywhere from YouTube or Wikipedia so I

explain the basic task of API to fetch

the link or to face the data from the

website okay so we have already

generated the API now let's see how to

use that API through esp8266 now let's

come back to the code okay so up till

here is the necessary declaration of

this TFT module which is necessary to be

declared before using this and this is

the host name and password

for this we need to provide for

connecting our esp8266 to a Wi-Fi

network so my host name is SMS and my

password is 8 times F then the functions

need to be performed

so I already perform the functions like

gold and temperature and I don't think

when I use I've also performed string

use yes

so this is basically the collection of

the functions now let's start or let's

start with a new function name string

time let's say we will make a function

of time and which will return a string

in response

so this or our declaration and setup I

have what you can say the baud rate is

set to 9000 sorry around 5 to double 0

and this is required to initialize that

watchman touch screen and the buttons

function this function will print the

buttons onto the screen for those who

have used this touchscreen module will

get easily understand so this all are

the buttons to be printed on the screen

so let's make a new button name a time

so I will just copy here we need to

provide the coordinates and the size of

the rectangle so I am simply copying the

size and changing the coordinates to 180

and this string or the data to time and

that this is yellow and I think the data

is sufficient ok let's upload and see up

with this a button should be appear

should appear on the screen naming name

time ok so ok ok ok I need to change the

coordinate here also so it will be 90

180 okay set cursor I need to change to

200 the y-axis is 200 okay so okay so

button name time appears now next after

buttons let us see what is there okay

after button a school goes into the loop

until here are the necessary coordinates

or 2 for the touch touch coordinates ok

so anywhere we whenever we touch the

screen it will go inside this function

and the coordinates will be saved in

integer X and integer Y so coordinate

and it will be show the coordinates

would be shown by the this speed or

sorry X&Y on our serial monitor we can

see the coordinates and according to the

coordinates we need to assign the

coordinates for particular function for

example for the reset button the

coordinates of X are 320 from 226 to 320

and for wife is from 82 to 129 so

likewise we need to assign the

coordinates so whenever we touch in a

row in this limit it will automatically

call a function reset same way we need

to make the coordinates for time I've

already assigned the coordinates for

time and we need to just copy and paste

another function because the pro the

what is at the command remains the same

only the name change this so we can just

simply copy and here it will black out

the rectangle above the buttons so it

will basically the clear what is printed

before and if deeper then it will show

time whenever it is touched I'm sorry

time and cursor zero comma zero wide

exercise three and fetching data and

call a function named time instead of

views but this is this much amount of

functions or this much command will be

executed whenever we touch in this

coordinates that is the time button so

now we need to just make a spring or

come I'm sorry we need to make a

function named time so we will make a

function here the spring time okay so

here also I will simply copy and paste

because the procedure is same changes

only in the API key only in the API

and the name so I will copy up till here

copy and paste it in this function okay

so I will discuss the instructions one

by one okay okay so before before that

we need to see what happens when we

press the reset button of each of which

commands are executed when you press

this reset button so I have made a

function Carol to reset I will discuss

that okay said he said so it will first

of all reset this Wi-Fi module with

80-plus rst and will disconnect any

disconnect the Wi-Fi through which is

all which the Wi-Fi module is already

connected so it will disconnect Wi-Fi is

connected and it will select the mode 3

that is both access point and station

mode and it will search for the Wi-Fi

network and will connect with the

network whose host name and password you

have provided so let's see what happens

when you press the reset button so what

he said ok so it shows resetting ok mode

with is equal to 3 and searching for

Wi-Fi network it will show connected

when it connects through the network so

stream is provided mmm come on come on

ok so this all our functions when

performed when the reset button is

pressed so after connecting connected

after the device get connected to the

Wi-Fi now the now we need to start from

here ok so will after connection

we will stop after connecting to the

network we will establish a TCP

connection so for fetching the data we

need to first of all establish a

connection through a command it T plus C

IP start so we need to send a T plus C

IP star is equal to the type of

connection it is TCP connection the host

server name and the port number port

number is here of course ad as it is

through HTTP ok okay so we can provide a

host sir

were named now the host server name is

basically the name of that host who is

aap isolated here it is API dot thanks

fake calm so I have made the string ATC

F star is equal to inverted comma TC be

inverted comma close comma inverted

comma if they are good things become

inverted form ax comma 18 and slash

lesson for enter

I am sending this string through send

data function of society data function

requires requires the string the timeout

period is 3000 that is 3 seconds and 0

and 1 if it is 1 it will show the

response on to a serial monitor that is

for debugging so I return 0 year so

after sorry as soon as we send this

command it will automatically ask us to

say ask us to provide the number of

characters we need to send which is 80

plus CI percent is equal to 90 so here

we are sending a 90 characters after

this this command so we need to provide

number of characters here and this will

be remained constant because in all the

different or different different API is

from things be calm only the API key

will be changed and the rest of all

things will remain same so the character

also will remain same as the key is

constant of 16 characters I think okay

so so this also remains constant the

change is only in this key now keep

generated is whichever key is undeterred

if you had things to do okay so we need

to just copy the key copy and paste the

key your control V okay so this is a now

we need to change the string to the IME

time okay so it will send this string

first of all the get API API key and the

host server name that is AP I thought

things picked odd form okay let's see

I'm just commenting this steps to filter

the data because before that you must

but must be aware that

which datum we are getting and why we

need to filter so we'll just make it one

so it shows the response in the season

monitor now I think control you upload

ok need to change here T IME time and

upload ok so let's see what data we are

getting and why we need to filter that

data ok so as soon as I press the time

but only source to fetching data ok ok

so here is the receive data are now

receive data that is IP D comma 11 comma

11 18 13 am closed ok out of that we

will require only this much amount of

data so we need to filter this data and

rest of the data are not at all useful

to us we need to filter it so that's why

I returned the code for filtering the

data and this is also very much very

much easy let's let's see the logic

behind this filtering so every time

whenever it gives response it will show

it will use plus IP D comma the number

of characters in response and then

semicolon and the data will start so we

required the data after the semicolon

and the data before this closed that is

the C character so I have made a

algorithm which will filter from a

filter the data starting from vertical

character and ending at particular

character so here our data is starting

from a semicolon and ending before

ending at C of C for closed so character

before require data starts and collected

after end require data ends ok so every

time is not the case that the data is in

between semicolon and the closed

function it may be between any of the

characters so you just need to change

the character before the starting of the

data and after the end our

after end of the theta so just provide

the characters and the data will be

filtered now let's see if this works or

not let's upload this okay so let's see

when I press the time button

it must works hope it works

okay so required data

we got the required data that is along

25 0 9 a.m. the delay is because of the

my screen recorder pressed so that is

the delay so any of the required data or

the required task is completed we face

the data from the website called time

and date calm and displayed on to our

touchscreen module or you can use the

data anywhere you want you can make even

a Wi-Fi of Wi-Fi clock which has the

data every 10 minutes or every second

every 5 second or 10 second and you want

so this is all about precision you can

fetch any data from any website like the

stock market data the Live Gold rate

which I have done like temperature

lifetime like YouTube views anything you

want let's make an API and making API

becomes much much much simpler using

things big calm so explore that website

the great website for IOT related things

and stay tuned for next such amazing

tutorials until then goodbye thanks for

watching