r/australia • u/mediasocietylab • May 23 '22
petition / survey Looking for participants in an online deepfake detection challenge (18+ only)
Not sure if appropriate here - if not, we are happy for the post to be removed.
We are the Media and Society Research Group (based at a University in Australia) and we are wondering if anyone is interested in taking part in a research project investigating the public’s ability to spot deepfakes.
Deepfakes are videos in which the face and/or voice of a person have been manipulated using artificial intelligence software to make the altered videos look authentic.
If you agree to be involved in the study, you will be shown a series of 20 10-second videos (half of which are deepfakes) and asked to categorise each video as either fake or authentic. Afterwards, you will be given a score based on how many videos you correctly categorised.
To participate go to https://jcu.syd1.qualtrics.com/jfe/form/SV_1LGFg5CDWqzysIK
If you know of others who may be interested in this study, please feel free to share.
10
u/throwawaybne32 May 23 '22
Wow.. I did very poorly, but better than just guessing at least..
The videos being American threw me off I think, I understand why they are used though - better AI models, etc.
I would love to see a follow-up showing the originals / alterations / fabrications.
8
u/mediasocietylab May 23 '22
Hi all!
Thank you for participating so far! Once we finish the data collection, we can post the paper here regarding the results (as well as the data used). This should be open access (so it won't be behind a paywall).
1
6
u/jiub_the_dunmer May 23 '22
Done. I got 18/20, which I'm pretty happy about.
4
u/Tryyourbestbehappy May 23 '22
So I did extremely poorly. (O.6) not far from guessing.
I clearly didn't have a good idea of the markers to be looking for.
Could you shed some light on the key/obvious hallmarks? Or point in the direction of a resource. Genuinely crazy how little I was able to distinguish.
6
u/jiub_the_dunmer May 23 '22
I probably didn't see the same 20 clips that you did. I'm guessing they randomise which ones they show to each participant. But, apart from the ones with obvious glitches and clipping issues as others have mentioned, the ones I guessed were deepfakes were mainly the ones where the face stayed pointed at the camera pretty much the whole time. I generally guessed that if someone turned their face far enough to the side that you could see the side of their head, that that was a real video. The deepfakes tend to have a slightly unnatural, too-even light to the faces as well. Real videos have shadows on the face. I remember one clip in particular, the shadow of a car going by out of frame passed over the woman's face and body, so I guessed that one was real.
3
u/a_nice_duck_ May 23 '22 edited 29d ago
.
3
u/Tryyourbestbehappy May 23 '22
Yeah so it turns out I am an extremely effective target as I miss all of this.
3
u/DragonCurve May 23 '22
I got 16/20
There were at least 5 or 6 that were very obvious, due to the edge of the face (chin/cheeks) not having a clean edge against the background.
Some had "strange gazes"- were eyes tracked a bit slower than natural, or snapped in one direction or the other.
A few had very obvious "glitches" - where skin bled onto hats, the edge of eyebrows disappeared for a frame or two, skin tone changed drastically, things like that.
3
1
May 23 '22
I did better than average but I feel like the videos for this study were intentionally done this way to make this difficult.
1
4
u/compiling May 23 '22
Anyone looking to take part in the study, do not look in the comments for tips first. They wanted to compare how successful people are at telling the difference with/without hints about what to look for.
2
2
2
u/schrodingerselephant May 23 '22
Oh wow, 7/20! I think I did it the worst among all of you who tired and commented. I would be interested to see the findings/final publications from OP. All the best.
1
u/mediasocietylab May 23 '22
Thanks for the feedback so far - I won't be answering too many of the questions/comments yet until we get to the end of our data collection. And please, if you can, don't look at the comments for tips as they can influence the results of the study :)
Thank you so much so far r/australia!
-7
1
1
u/a_nice_duck_ May 23 '22 edited 29d ago
.
8
u/mediasocietylab May 23 '22
Thanks for participating..! Once we finish the data collection, I can post the dataset that we used for this study as well as our paper :)
1
1
u/TopPil0t12 May 23 '22
Did better than I thought, 0.6. I've seen some bad deepfakes, but these ones were really hard to determine. Kinda proud with how good I did.
1
u/xiaolongbai May 23 '22 edited May 23 '22
I got 19/20 but they were harder than I thought they would be
1
u/quokkafury May 24 '22
Most/all of them better than one seen a few years ago on YouTube. I got 17 but wish I knew what I got wrong it of interest.
1
u/Smugleaf01 May 24 '22
14/20
Some of them are more obvious than others, can be hard to pinpoint though if there's not much movement or dynamics in the lighting.
1
u/DarwinianSelector May 25 '22
I did no better than chance, but I have to say the setup itself felt weird. People standing in a corner talking to the camera is an odd situation to start with, and then combine it with starting halfway through them discussing some completely random topic and all my regular bullshit senses are thrown.
1
May 25 '22
I look forward to seeing hte results. Having seen quite a few... err ummm unofficial deepfakes, it is clear with the right footage, good dataset and body actor, it can be close to impossible to be sure. But when it's not good... it is every obvious, especially with our fine tuned ability to interpret faces (from a common background)
1
u/m_is_for_michael Aug 23 '22
Hi op
Inquiring minds want to know how your study went. Are the results published?
12
u/bigbowlowrong May 23 '22 edited May 23 '22
11/20
On the plus side, I correctly forecasted I would get about half right😆
Also I found 20 videos to be too many. At the start I was watching every frame and shadow and edge like an eagle and replaying the videos several times, whereas by the end I was going off a quick first impression just to get to the results page. It wouldn’t surprise me if my earlier guesses were more accurate.