holy fuck my parents nearly got scammed by one of those AI voice calls today. They used MY VOICE TO MAKE MY PARENTS THINK I ACCIDENTALLY KILLED TWO PEOPLE WITH MY CAR AND THAT THEY NEEDED TO PAY MY BAIL
Comments
Log in with your Bluesky account to leave a comment
Oh my god I didn't know it had gotten to this point already that's terrifying - I just discussed this with my mom after reading this. There's something so... invasive about the idea of my voice being cloned I'm so sorry that happened to you
Every news story about this can be traced back to the same FTC press release, which is literally just "hey, maybe scammers could use AI voices to run the grandparent scam? better watch out!" Nobody has a recording, AI voice generation on the fly is way worse than a human faker, etc.
Ever spoken during a scam or silent phone call beyond saying “hello”? Definitely think on the public connections or non-govt website databases (in the event of a breach - 23& me had a set of breaches from users reusing passwords) between you and your folks.
Ordinarily Occam’s razor is the way to unravel the source of scams: do you have a definitive list of places where your voice has been posted online (say YouTube or TikTok)? Then there’s the question of linking you to your parents, and what if anything made your parents special enough to target
The more parsimonious explanation is that there was no fake AI voice, which eliminates the need for all of that other stuff. Parents and other family members have been falling for this scam since long before AI voice generation was available
I’m definitely inclined to agree. Our minds are plastic and we skip pretty glaring details in favour of a narrative. I’d need compelling evidence before I was convinced of an ai voice attack, especially at this level of cybercrime.
i just texted my mom about it and i thought it was a new scam but apparently it’s been a thing since at least last year 😭 i had no clue. the future sucks lol.
Whenever I call my cell phone provider, I am asked whether I agree to the conversation being recorded. I have no doubt that we all have voice samples somewhere that can be used to train AI.
The alarming thing about this scam is that it catches everyone, but for people with elderly folk in their lives it can get blown out of proportion quickly. The recommendation is to have a codeword or phrase to confirm it's you.
One time a dude got a call from my number and I woke up to a voicemail that just said “dad?”
I texted the number I got the voicemail from and we both shared our experiences with scammers and went on our way. Both of us now know they can use your number.
my mom kept one of them busy once for about a half an hour-- her "grandson" was in for some kind of drug possession when she only had a four-year old granddaughter at the time
called her everything under the sun after she finally told them to go fuck themselves, one of her favorite stories
Hi!
You probably only need about 5-10 minutes of data to clone someone’s voice, considering the quality is only over the phone.
I don’t think there’s much anyone can do aside from lobby their local government to make this shit a crime.
For proprietary and private software they probably only need that… which wouldn’t give great results on accent mimicking
I’m almost certain the scam would use the publically available and free RVC software which I’m familiar enough with to estimate 1-3 minutes at least
Oh wait I get it now sorry
Yes scams have gone on for ages
But this mutation is pretty dangerous because many people aren’t familiar with the way it works
I could whip up an ai deepfake of your voice in a day with a few minutes of data.
Where do you think the ai covers “Nicki Minaj sings the national anthem” type shit is coming from
OP is incorrect. They did this to my parents before that was possible. It's a classic con; you let the target mentally adjust to what they expect and get them to fill in the details for you. You don't need to know who anyone is.
My grandparents lost $2k to this kind of scam. It was before voice tech was at this point, though. The scammers just put a guy on the phone that claimed to be my brother and they couldn't tell the voice was off
God I'm so glad that the bank caught it and were able to protect them but that's so so fucked 😬 I'm sorry you had to deal with this, it must be extremely unnerving. Talking to my family about it now.
nah they've been doing this one for ages, it's a con they don't need to do your voice or actually know anything about you
(source: they tried it on my parents absolutely before AI voice synth was possible)
like seriously how would they train it how much footage of you talking is online
Thanks for the heads up. My dad was recently targeted by similar schmucks but this is even worse. But telling people about it in advance may help. Glad your parents weren’t scammed.
luckily LUCKILY the person at the bank spotted the scam and insisted they call my cell just to make sure. God how fucked is our fucking world that I need to start using a password to talk to my mom and dad on the phone
I've instructed my mom that if anyone calls saying I'm in the hospital or something ask them for the name of my first dog. Yes, that's what it's come to because of these scams directed at seniors.
Omg my aunt and grandmother almost fell for a similar one. Only they claimed to be a public defender calling for my brothers-in-law. My aunt got all the way to the bank when my sister returned her call just in time.
how did they get my voice I’m trans I don’t LIKE my voice. I don’t put it on the internet as much as I’m able. I’m willing to bet microsoft and discord and zoom and apple are all scraping calls and selling voice data these days aren’t they
Ugh. I get a lot of spam calls and almost never answer, but there are a lot where no one ever says anything, so you just go "Hello? Anyone there?" and they sample your voice that way, I'll bet.
I don't know for sure but my guess is when they get a hit good enough to sample, they start looking for other phone numbers related to that one (not hard to find) so they can call parents or other relatives.
I would love to hear them try to make a speech that would fool my family using my Walter-Matthau-pretending-to-be-Eleanor-Roosevelt voice. "Hellooooieu?"
There was a demo at a recent security conference where they were able to copy someone’s voice from just three words. Like “hello? Anyone there?” was enough.
Thanks—I ask because your parents were put in an incredibly stressful situation and so their judgment of whether it was you might have been compromised. I think it’d be easy to do this scam in bulk w/o any real voice data.
Yes, my father was scammed this way about my brother, same bail request. I have no doubt there’s weird recording stuff going on, but in his case i think he heard what he was primed to hear.
Fortunately my mom was never taken in by scams, and would answer the "Microsoft Windows Repair" ones by telling them she had a Mac, and they didn't ever get far enough to call back as Apple repair and get told her Mac was on dialup :-)
Had somebody tried the bail one, she'd have asked which kid.
I’m in Metro Detroit and a few weeks ago they did a piece on this on our local news. It sounded pretty convincing. The cadence was a little odd and the AI was trained on our AG and one of the anchors so the program would have had a lot to work with. But still.
Discord isn't scraping those to send to advertisers to be clear. It's scraping that for other users to record.
AFAICT Discord can't sell the data it does have because it's not linked to anything outside Discord and Discord itself is ad-free. If they had ad sales by now they'd probably be profitable
A lot of them are dialing a ton of numbers at once because they know most people won't answer. There's not always an actual opening in the queue when someone does.
This went from "literal plot point in a fantasy K-drama" to real life in like sixty days, Jesus Chris. I'm glad they were able to reverse or prevent the transfer at least.
Having a code phrase is a good idea. In a similar way, if you go to somewhere dangerous and (usually) your employer takes K&R insurance (that's Kidnap & Rescue), you set up one or more proof of life questions that only you and someone close to you know the answer to.
Good on the bank for keeping their employees well-trained. They are trained to decline when elderly people suddenly want large sums of money ("doesn't fit the previously established customer behaviour" is one signal of a customer possibly falling victim to a scam).
Wow. For once USA can be happy that they are ten years behind the rest of the world when it comes to banking. Here in Norway, the money would have just been transferred, with the phone, while they were talking to the scammers on the very same phone
I used to work for a company that processed data from Apple’s Siri. The job involved listening to hours and hours of people talking to their Apple devices, or sometimes just in proximity of their Apple devices.
It becomes tough to tell what actually IS one of those things and what isn’t, if you’ve bought any electronics at all in the last five years or so, like a TV that lets you enter by voice search phrases.
Seems a rather dubious feature that almost certainly is there to mine data.
Scraping and selling anonymized, aggregated voice data without notification or consent would be bad enough, but if the scammers were able to connect your voice samples with your name and your parents’ phone number, that’s so much worse!
The spam people with bogus scams and hope one in a million are gullible enough to fall for it is a common scam strategy, but this sounds like something more personal.
August. Actually March… August was just when the potential issue came to light. It was a big deal on Twitter for a few days. Honestly I didn’t follow the controversy too closely at the time but a web search will yield a lot of articles. This summary seems good. https://www.cbsnews.com/news/zoom-privacy-issues-user-agreement/
Discord has started rolling out Clips, switched on by default. it allows people to record your voice in voice chat. Alert I saw said you would get no notification you were being recorded
Turn off in settings if it has been rolled out to you
Scammers scrape voices from TikTok or other social media… they only need you to say a minimum of 17 words and they can do a pretty great job of mimicing a voice.
If you don't talk on tiktok or whatever I would check your phone and pc for Trojans and malware. It's possible they got access to your phone and contacts through that.
Another idea might be a fake zoom call - had any zoom dates with any hotties? Any interviews with companies that turned you down?
FUCK this is my worst nightmare. This is why I never answer spam calls anymore. This is also why I'm tempted to delete my voice mailbox and let it revert to the robot.
They didn't get it, it's just a reasonable sounding copy. Especially when they're pretending to be upset.
Mexicans will get calls from 'family' who need money to 'pay coyote.' They say he'll abndone them if they don't pay up. So grandma sends money. 🙄
Weird to see the scam jump to other targets.
I think the bail version is relatively common regardless of race/ethnicity. Pretty sure I've also heard of car trouble or medical bills as the pretext. Evil business regardless.
The border money scam has been going on for years. They specifically prey on migrants because they're less likely to go to Johnny law when they lose money.
A private eye told me to only talk to people on the actual phone because there lots of laws governing old school phone calls. You’d need at least a FISA warrant to tap them where tech cos can do whatever with your IMs and Slack huddles
I'm sorry
What?
AI voices for me were text to speech claiming to be from Amazon. They were in inform me that my prime subscription was being terminated.
I don't own a Amazon prime subscription
Man you're like the third person in comics alone I know of that this has happened to. Over Xmas I warned my folks about this and told them to hang up and call me back if it ever happened. AI voice shit sucks asssss
It doesn't have to be AI. Ten years ago they tried to hit my grandparents and they used a crying/screaming voice to distort natural sounds. Also generic terms. I'm sure AI could be involved, I'm just saying this is older than that and doesn't need to be.
I run the Privacy and Cyber Crisis Advisory Firm Black Alchemy Solutions. Please feel free to watch our videos on IG @@blackalchemysolutions and on YouTube. The videos give real life scam scenarios that are informative and easy to digest
My grandpa got a phone call like this with my brother’s voice- he was on the phone with his bank to transfer money when his (incredible) companion heard and realized it was a scam. He didn’t believe it (voice was so real) till she got my actual brother on the phone.
holy crap, that's terrifying, that they have your voice and how much info they have. Do you, by chance, have a smart speaker? I wonder if that's how they get it.
I heard recently that we all should have a code word with our loved ones so they can ask scammers like this for the word if it sounds like one of us. I can’t believe this is where we are now.
My grandma nearly got got by one of those, but was saved by the fact that a) she didn't use online banking and b) she didn't drive. So she physically could not give the scammers the money but it was still incredibly distressing to her.
Yeah thats freakin' scary. Older people are such a big target for those sorts of scams. Even before the AI boom I used to get them all the time when I worked tech support... -_-;
scammers are not using AI, it’s still the same generic woman screaming/crying but people believe their parents when they say “oh honey it sounded EXACTLY like you!”
Has there been any proof reported that it’s actually AI? Seems unlikely.”
Had a call like that, I kept asking verification questions & they finally hung up on me🤣🤣🤣
How old are you now, where are you, where do you go to school, what’s my name ( besides ‘Hi Grandma)
Holy shit, I'm so glad the bank employee caught it. Anyone and everyone can get scammed, they're only getting smarter. I even fell for a different kind right before Xmas. Ego bruised but lesson rightly learned. Very important for folks to be open about it!
I feel like this sort of thing might kick off a massive class action lawsuit, because if a company secretly/illegally collects voice data and is selling it without the original users knowledge or consent......
Several years ago there were models that could clone voices with 5 seconds of audio, last year Microsoft got it down to 3 (though afaik that model wasn't publicly available).
I'm sure they're not perfect, but it's still pretty awful.
I remember the radio lab episode that covered this. Microsoft pulled the plug because they didn’t think it should be available publicly after receiving some criticism (rightly).
This is so scary. I took this as a sign to set up an agreed-upon word with my parents so they'll know that it's me if there's ever an emergency. So glad the credit union caught it.
Comments
this is an old con
there's no AI, just expectations adjusting perception
a whole system to make people afraid of the unknown, of anything new, anything different from them
What people actually do with ai:
I texted the number I got the voicemail from and we both shared our experiences with scammers and went on our way. Both of us now know they can use your number.
I hate this timeline.
called her everything under the sun after she finally told them to go fuck themselves, one of her favorite stories
You probably only need about 5-10 minutes of data to clone someone’s voice, considering the quality is only over the phone.
I don’t think there’s much anyone can do aside from lobby their local government to make this shit a crime.
I’m almost certain the scam would use the publically available and free RVC software which I’m familiar enough with to estimate 1-3 minutes at least
They can use it from within a discord bot by the way
It’s very accessible
Yes scams have gone on for ages
But this mutation is pretty dangerous because many people aren’t familiar with the way it works
these are done blind
it's just con skills, not AI, which would take vastly more work and knowledge
Where do you think the ai covers “Nicki Minaj sings the national anthem” type shit is coming from
(source: they tried it on my parents absolutely before AI voice synth was possible)
like seriously how would they train it how much footage of you talking is online
I had to hang the conversation with someone once and they didn't want to listen when I said not to trust the original call they got
I would love to hear them try to make a speech that would fool my family using my Walter-Matthau-pretending-to-be-Eleanor-Roosevelt voice. "Hellooooieu?"
https://bsky.app/profile/almost-elly.bsky.social/post/3kioh37z3ys2l
The default should never be Opt in, it should ALWAYS be Opt out.
https://bsky.app/profile/almost-elly.bsky.social/post/3kioh37z3ys2l
I still suspect they could also get it from a cold call where they say nothing and then hang up.
https://www.msn.com/en-us/news/us/family-member-calls-for-help-it-could-be-ai-scam-tn-sheriff-warns/ar-AA1mPOAV
I'm so sorry this happened to the OP's parents.
Had somebody tried the bail one, she'd have asked which kid.
AFAICT Discord can't sell the data it does have because it's not linked to anything outside Discord and Discord itself is ad-free. If they had ad sales by now they'd probably be profitable
https://www.msn.com/en-us/news/us/family-member-calls-for-help-it-could-be-ai-scam-tn-sheriff-warns/ar-AA1mPOAV
From what I have read Clips gives a notification when in use. So far I have not seen anyone use it.
So I just wouldn’t be surprised.
Seems a rather dubious feature that almost certainly is there to mine data.
The spam people with bogus scams and hope one in a million are gullible enough to fall for it is a common scam strategy, but this sounds like something more personal.
I teach on there and sessions get recorded for students.
Turn off in settings if it has been rolled out to you
Another idea might be a fake zoom call - had any zoom dates with any hotties? Any interviews with companies that turned you down?
Mexicans will get calls from 'family' who need money to 'pay coyote.' They say he'll abndone them if they don't pay up. So grandma sends money. 🙄
Weird to see the scam jump to other targets.
What?
AI voices for me were text to speech claiming to be from Amazon. They were in inform me that my prime subscription was being terminated.
I don't own a Amazon prime subscription
I heard recently that we all should have a code word with our loved ones so they can ask scammers like this for the word if it sounds like one of us. I can’t believe this is where we are now.
scammers are not using AI, it’s still the same generic woman screaming/crying but people believe their parents when they say “oh honey it sounded EXACTLY like you!”
Has there been any proof reported that it’s actually AI? Seems unlikely.”
https://www.reddit.com/r/Scams/comments/12n225h/is_there_any_proof_to_the_claim_that_ai_voice/
How old are you now, where are you, where do you go to school, what’s my name ( besides ‘Hi Grandma)
How did they replicate your voice???
I'm sure they're not perfect, but it's still pretty awful.
“Oh come on, sweetie, it was just one victim. How much could the bail be? You can’t handle it yourself?”
SO I GO "HELLO? HELLO?" INTO THE VOID AND THE MACHINE TRIES TO CAPTURE ENOUGH SAMPLES THAT IT CAN REPLICATE MY VOICE???
....AAAAAAAAAAAAAAAAAUGH. JEEZ. AUUGHHHhh