cheat

Can AI Proctors Detect Online Exam Cheating? | Automated Online Exam Proctoring



Sharing buttons:

it's final season and many of you are

probably studying for your exams right

now

in fact i have to go back to studying

for mine once i finish filming this

video

but in an era of online learning our

exams may look a little

different instead of using tiny desks in

lecture halls or sitting in classrooms

or getting more take-home tests and

online exams and with that has come an

increased concern about cheating

after all there's no proctor or teacher

to watch you take the test

right well several companies have

developed ai-based solutions to those

concerns

from monitoring your behavior through

your webcam to tracking your keystrokes

these companies claim that they can tell

whether or not you're cheating during

exam using

automated remote proctoring but how true

is that and how does it even work

let's find out if you're new here i'm

jordan and welcome to the wonderful

world of artificial intelligence

consider subscribing if you're

interested in learning more and smash

that like button so that i know that you

enjoyed this video

also let me know in the comments if

you've taken an exam with online

proctoring ai based or

otherwise all right let's get into it so

online proctoring has actually been

around for way longer than i thought

when i set out to make this video around

10 years now and the original approach

was pretty straightforward you would log

into your exam using the online

proctoring software

and they would connect you via video to

a remote proctor who would watch you

take the test

ideally they would be familiar enough

with the exam to be able to answer any

questions you had about it but

just like an in-person proctor they

primarily served as a cheating deterrent

subsequent versions of this software

incorporated the ability to track you

as you took the test from monitoring

your video to monitoring your online

activity

are you googling the questions did you

just copy and paste something into your

essay all that activity would be visible

to the proctor who would use it to

determine whether or not you were

exhibiting cheating behaviors

from there it wasn't a huge leap to

automating the role of the proctor

after all the software already records

and monitors all of your activity

so the next step would be to train an

algorithm to detect whether or not an

activity would be considered cheating

other functionalities including facial

recognition and location tracking were

also added to make sure that you were in

fact the person who was supposed to be

taking the test

and some companies even offer audio

analysis to make sure that you're not

talking to someone else off screen who's

feeding you the answers

as well as a manual offline human review

to catch any errors that the algorithm

may have made

and now there are several companies that

offer these services predominantly to

universities they've been picking up

steam since well before coronavirus

forced everyone towards online learning

after all if you've ever been in a huge

lecture something like psych 101 at

cornell that has like a thousand people

in it

you know that it's functionally

impossible to determine whether

anyone is cheating because there's so

many students and so few teaching staff

and proctors

additionally exams are often offered in

the evening so

for commuter schools it may make more

sense to offer an online exam so that

students don't have to either stay late

on campus

or return to campus in the evening in

these scenarios automated online

proctoring is extremely useful it takes

the burden off of teaching staff to have

to keep track of all of the people

taking the test

can simultaneously monitor hundreds of

students

and allow students who don't necessarily

live on campus to take the exams from

home

but how does the software tell if you're

cheating well these are all proprietary

algorithms so i obviously can't pull

back the curtain

to show you but there are two

overarching strategies that i've seen

used first up is identity verification

or making sure that the person who's

supposed to be taking the test is

actually the person taking the test

this is often done at the beginning of

the exam usually by comparing your face

in the video feed to

a photo on record from your student id

but can also be done throughout the exam

second is activity monitoring this

includes anything from eye tracking to

making sure that you're focusing on the

screen

to disabling copy paste so that you

can't copy something from outside of the

exam

into it to analyzing your audio to make

sure that someone else isn't telling you

the answers

anything deemed by the algorithm to be a

suspicious behavior will trigger an

alert

either for the instructor for the

student or both the outcome of that

alert could range from anything from a

human review

of the alert itself to disciplinary

action on the part of the instructor

in short the software detects cheating

by being trained either through

simpler algorithms such as pattern

matching or through more complex

algorithms like deep learning algorithms

to track your identity and your activity

throughout the exam

and make a continuous prediction as to

whether or not you are cheating

now you might be wondering how an

algorithm decides whether an activity is

suspicious

the answer to this question honestly

deserves its own video as past work and

machine learning has shown that it's

very hard to make a training set that

doesn't have bias to it

and it can be very hard to extrapolate

the results of your model

to the real world as an example i'll

include a link in the description box to

a pre-print that tried to detect

deception using gate the authors faced

considerable criticism when this

preprint was originally uploaded to

archive and they've actually since

updated it to address some of the

limitations of their technology a bit

better

because the training data that they

generated was extremely contrived and

didn't seem to have a clear real world

parallel

and in fairness this is a challenge for

most of the work in this field because

determining the intent behind an action

is difficult for

humans let alone algorithms so the

software likely relies on a set of

training data that has been labeled

where some examples represent cheating

and some

do not it's hard to know more past that

now let's jump back to the question in

the title of this video

how well can automated online proctoring

identify cheating behaviors

obviously most of these companies as

well as the schools that use them claim

that they can identify cheating

behaviors better

than you can in normal in classroom

exams as well as in online exams i would

believe that claim to be true but would

also

question any accuracy statistics that

come with it because

it's unlikely that you can reliably

quantify how many people actually

cheated on a test more broadly i think

that online proctoring is something that

we're all going to have to get

comfortable with going forward

i don't think it's universally good or

bad after all there's a reason why we

normally have people proctoring exams

when we take them in person so it

doesn't surprise me and it does make

sense that you would have some

analogous system that can perform that

for online education

and automated online proctoring is

particularly useful right now when we

literally cannot

have proctors in the room with students

however it also has faults

students of color have complained that

the facial recognition system can only

identify them in specific lighting which

you may not have

access to wherever you're taking your

exam and these systems can require more

bandwidth than

students may have access to if you can

only take your online exam with the

library you might not be able to

download the necessary software to even

take the test

granted a lot of these things are also

challenges for online learning overall

there have also been some concerns about

data privacy after all the system is

recording while you take a test

and in the us we have a federal law

called ferpa which basically requires

that universities

maintain the privacy of your educational

records but i haven't found any claims

of data mismanagement or misuse so far

and it does seem like these companies

are doing their best to comply with

ferpa

now i'll say i personally haven't used

any of these systems in any of my online

exams so far

i have one final this semester and while

it is online as far as i know

it is not monitored in any way but in

short i think a lot of the work that

an ai proctor does in detecting online

cheating

isn't actually in the algorithm itself

it's in the psychology of having

someone or something watch you while you

take a test

and while this version of proctoring may

be different from what we're used to

it's likely not going away anytime soon

so you definitely shouldn't take it as

like a

challenge to fool the ai or anything

it's

not worth it if you want to learn more

about how artificial intelligence is

used in schools you can check out this

video up here

and if you'd like to subscribe to my

channel then you can click

right here to do that if you'd like to

learn about my phd life

you can follow me on twitter and

instagram thanks as always to all of my

patrons and i will see you guys next

friday

bye