Ep. 176: Extracting Audio From A Digital Photograph + When A Scientist Investigated His Own 'Haunting'

This episode explores the feasibility of extracting audio from digital photographs and recounts a personal investigation of a haunting at a workplace by a scientist.

1. Smartphones can inadvertently capture audio through image stabilization mechanisms in their cameras.
2. The technique developed can pinpoint voices in a photograph with up to 81% accuracy, raising potential privacy and security implications.
3. The episode also highlights a real-life scenario that resembled a haunting but was scientifically explained through the influence of infrasound.
4. This story exemplifies how scientific curiosity and investigation can demystify seemingly paranormal experiences.
5. It emphasizes the importance of considering environmental factors in workplace design to prevent discomfort and fear among workers.

1: Introduction to Audio Extraction

Jill introduces the concept of extracting audio from digital photos, describing how smartphone cameras can pick up sound vibrations. Jill ChaCha: "Turns out your smartphone's lenses are inadvertently picking up audio information too."

2: Scientific Discovery and Application

Details the scientific process and potential applications of the audio extraction technology, including its implications for legal scenarios. Jill ChaCha: "This could change the world for the better, especially when it comes to false imprisonment and shoddy justice."

3: The Haunting Investigated

Recounts the tale of Vic Tandy’s investigation into the supposed haunting at his workplace, which was eventually attributed to infrasound. Jill ChaCha: "It felt like those wildcats were moving around, and the groans and creaks from what was now a deserted factory were spooky."

1. Consider Environmental Health: Always evaluate your working environment for factors that could be affecting your health and perception.
2. Embrace Technology with Caution: While new technologies can provide amazing benefits, consider privacy and security implications.
3. Stay Informed: Keep up with technological advancements to understand how they might impact your daily life.
4. Scientific Inquiry: Use curiosity and scientific methods to investigate unusual occurrences.
5. Prioritize Safety in Design: Ensure that workplace designs consider all aspects of employee well-being.

Hold onto your cheeks. Things are going bump in the night and revealing what you said when you took that questionable selfie alone. That’s right, there’s more than meets the eye.

People

Vic Tandy

Companies

Northeastern University

Content Warnings:

None

Jill Chacha

Why do it?

Hey, welcome back to well, that's interesting. The Ghost in the machine edition. Today is episode 176, extracting audio from a digital photo. Photograph and when a scientist investigated his own haunting. My friends, I am proud to say today's topics are equal parts creepy, curious, and fascinating.

They exist beyond the realm of human senses. And because of that, shit is going to get weird. It's going to feel weird. But once it's all explained, the answer, once again, is stranger than any fiction a human mind could pull out of its butt. If I may be so poetic, in the first half of the show, voices coming from a still image.

Yes, it's very possible. The photos saved on your camera roll not only reveal how many times it takes to capture a decent selfie, but they can also reveal what you said at the moment you snapped that photo. Or even what the people around you were saying at the time. And they didn't even have to be in the picture for their voices to be recorded. My friends, if this sounds like echoes exist in your digital pics, you're kind of right.

Turns out your smartphone's lenses, while aimed at capturing visual information, are inadvertently picking up audio information too, thanks to an interesting liquidy element of their design. Now, I'm not going to give away all the details here, but I can tell you now. An ingenious and real life Scooby doo team at Northeastern University took advantage of the image stabilization technology found in phones today and created a program that can hear who was around you while snapping a pic of that meal you ate two weeks ago. Are you intrigued? Horrified?

Me too. We'll get into it then. After the break in 1998, a group of scientists working in a laboratory at a medical equipment company began noticing and feeling strange things. Now, if you ever worked in an office, I'm sure you've felt depression, paranoia, anxiety, aches and pains too. But what this staff was experiencing was enough to make phds run out of the room.

After seeing a dark figure from the periphery of his eye. Oh, and watching one of his tools vibrate seemingly uncontrollably, one employee scientist took it upon himself to figure out what the fuck was going on around there. What exactly the team witnessed and experienced the research and the experiment to find out who or what was torturing them shall all be revealed. And no, smarty pants, it wasn't carbon monoxide. I'm Jill Chacha, and if this is your first time listening, welcome to the flock, my spooky business goose.

To begin, we need to pop into our freshly cleaned time machine and dial it way the fuck back to a time when the world economy was practically in shambles, the Middle east was on fire, and a presidential election was a point of contention for many. I'm talking about 2009. Now, we've briefly dabbled in 2009 before. In episode 161, recreating a pop song using only brainwaves, I mentioned that this was the year the cell phone matured into the smartphone. It was the year apps appeared and a record 1.73 billion people were using what was called the Internet.

It truly seemed like the possibilities were endless. It felt as though technology could save us. And mirroring these hopes, dreams, and promises were tv shows, especially those of the Sci-Fi genre. My friends, I'm going to ask you to pick up what's called a remote control and point it at something called a cable box. Thank you.

Now for our time traveling fantasy. We are in New York City, 2009, and we need to turn on the Fox network, which is channel five. So please hit the number five on the remote pad. Thank you. The show airing right now is called Fringe.

And for my fellow Americans who are of the X Files generation, like me, fringe very much follows the X Files tradition. This means with every episode, something seemingly paranormal or out of the extraordinary happens. And an underdog team of agents belonging to some outcast wing of the FBI swoops in to solve this mystery of the week. This week is no different. And the episode we're watching is called the road not taken.

Now, this episode is like 48 minutes long and has about 4800 things happening in it. So I'm going to give you a real short recap and only bring up the stuff that eventually inspires researchers living in 2023. I'm serious. So my dazzling business goose. According to tvtropes.org, today the fringe team is investigating the case of a woman who spontaneously combusts in the middle of a busy New York City street.

Now, as a New Yorker, I know this is totally understandable, but in tv land, it's bizarre. Long story short, the team discovers that the victim is the subject of a ZFT experiment to cultivate pyrokinesis. Of course. And from what I could gather, ZFT is a kind of terrorist organization. This was when the US was obsessed with the idea of outsiders destroying America.

If we only knew at the time the call was coming from inside the house. Anyway, continuing with the show as they investigate, Olivia, the main character, experiences visions while awake. And Walter, another character man, explains she is seeing an alternate universe which has branched off from our own. Now, in these visions, Olivia sees that the victim has an identical twin who is also subject to these experiments. And when they go and try to find her, all they could find among some rubble is melted glass from a window.

This evidence, my Sherlocky business goose, was not poo pooed away by our fringe team. Nay, they believe it's possible. Information lies within once they take this glass back to the lab. Peter, another character, quote, uses some ridiculous pseudoscience technique to extract sound from the glass it recorded when it was soft. This is very far fetched.

But they manage to hear a ringtone from the scene, which Olivia recognizes and calls bingo. Mr. Nasty himself. Sanford Harris answers, and they're on their way to finding the girl and getting some answers. End quote from denavgeek.com now, why he's called Mr.

Nasty, I'll leave that to you. I don't know. Right now we're going to focus on what Billy Grifter of Den of Geek called, quote, some ridiculous pseudoscience technique. And now we're going to pop back into our time machine and fast forward 14 goddamn years later to 2023. Yes, it's 2023.

The fringe show can now be streamed on Amazon Prime, Max and the tube of you. It's still adored by Sci-Fi fans and insanely smart nerds alike, including Kevin Foo, a professor of electrical and computer engineering and computer science at Northeastern University. He also watched the road not taken and read that same review, and he thought, quote, I bet we can do that. My lab specializes in the impossible. We usually expect the first reaction to anything we do to be, you can't do that.

And we say we already did. End quote. Fu told News Northeastern.edu now, Foo didn't focus on glass, but something else frozen and even more relatable and seemingly unimaginable to us smartphone obsessed humans. He and his team successfully found a way to extract sound from a still digital photo and muted videos. We need to get into this.

So join me, will you, at Foo's fantastic laboratory at Northeastern University, where our sleuthy team started with the obvious how a phone's camera takes photos. Well, my snazzy business goose equally. Long story short, nearly every dang smartphone has some kind of image stabilization technology that helps floppy hands avoid ruining those oh so special moments at drunken brunches. Inside your camera are small springs that hold lenses suspended in liquid, an electromagnet, and sensors then push the lenses in equal and opposite directions to reduce camera shake, my friends. Thanks to this and one other quirky design that I'm about to bring up we can hear anything juicy, Fu told a Northeastern's blog quote.

Whenever someone speaks near a camera lens, it causes tiny vibrations in the springs and bends the light ever so slightly. The angle of the light changes almost imperceptibly, unless you're looking for it. Normally, it would be hard to extract sonic frequency from those microscopic vibrations. But the rolling shutter, a method of Photography most cell phone cameras use today, actually makes it easier to achieve the impossible. The way cameras work today to reduce cost, basically, is that they don't scan all pixels of an image simultaneously.

They do it one row at a time. That happens hundreds of thousands of times in a single photo. What this basically means is you're able to amplify over a thousand times how much frequency information you can get. Basically the granularity of the audio, end quote. Fuck.

My friends, I think I know what you may be thinking. Mostly. Oh, God, how accurate is this? Like, can you really hear me talking shit about coworkers while taking photos on a beach vacation? Well, my paranoid business goose, good news.

Foo's team created a new technology that is. Drumroll, Drumroll, please. About 81% accurate. Holy fuck. Round of applause.

So what did they create exactly? God. My friends, may I introduce to you the appropriately named machine learning algorithm called sideeye and how it learns. You may be wondering. Don't worry, I've got you.

Well, fu got you. And explained everything to ifl science. If you want to know if I said yes or no, I can train side eye on people saying yes and no, and then look at the patterns, and then with high confidence when I get an image later, know if someone said yes or no, end quote. So, using this methodology and testing their system on ten different smartphones, Foo's team found that it could recognize spoken digits from a goddamn still image with 80.66% accuracy. Yeah, it gets better.

It could also identify which of the 20 participating speakers said the words with 91.28% accuracy. It could pick them out. And it could also guess the gender of speakers within 99.67% accuracy when two genders were taken into account. This is very accurate. This is very accurate.

So one might ask, is this a cybersecurity nightmare in the making? Perhaps. But the team is taking an optimistic approach to this technology. They believe this can change the world for the better, especially when it comes to the epidemic of false imprisonment and shoddy justice. Foo spoke with James Felton of IFL Science with an example of this quote.

Maybe there's an alibi and it's being admitted to court and somebody wants to prove someone was or wasn't there. You might be able to use this technique if you have an authenticated image with a known timestamp to confirm one way or the other. If you hear the person's voice, they're more than likely there. End quote. My friends, this is fucking wild.

Fu may have just single handedly created a real life fringe unit, and I'm sure it will only be used for good. After the break, an old fashioned ghost story that proves maybe we should all be working from home. That's right, an air quotes haunted workplace coming up next. Stay tuned.

Shopify

This episode is brought to you by Shopify. Forget the frustration of picking commerce platforms when you switch your business to Shopify, the global commerce platform that supercharges your selling wherever you sell. With Shopify, you'll harness the same intuitive features, trusted apps, and powerful analytics used by the world's leading brands. Sign up today for your $1 per month trial period@shopify.com. Tech all lowercase that's Shopify.com tech.

Kia

This episode is brought to you by Kia's first three row, all electric suv, the Kia Ev nine, with available all wheel drive and seating for up to seven adults with zero to 60 speed that thrills you 1 minute, and available lounge seats that unwind you the next. Visit kia.com ev nine. To learn more, ask your Kia dealer for availability. No system, no matter how advanced, can compensate for all driver error and or driving conditions. Always drive safely.

Jill Chacha

And we're back. We are so back. And my friends, I think we could all agree, offices are a scary place. The thought alone of having to make small talk sends deep shivers throughout my body. Pretending to care about your child's sports ball game this past weekend is basically the equivalent of a puzzle in one of those saw movies.

Anyway. Now imagine, if you will, for a moment, that your current or previous workplace isn't filled with dry snacks and ergonomic desk chairs. Imagine it's 1998. Sorry? Imagine it's 1998 and you're an engineering designer for a company that produces medical equipment, specifically equipment that focuses on intensive care and life support.

So you're actually surrounded by things like ventilators, kidney dialysis machines and defibrillators. They, of course, need to be tested and retested, so nearly every moment of your day you'll hear beeping or some kind of mechanical whir. Don't worry, it gets better. The space you're in is actually two garages bolted back to back, creating one long corridor that's 10ft wide and 30ft in length. At one end is the only entrance, a pair of doors that's closed at all times, and at the other end is a window and a cleanup station or washing station.

This lab of sorts only has three employees, and for one of them, a man named Vic Tandy, aka VT. This was his day job, and even though only two other engineers worked here, it was an office after all. Which meant, yes, gossip exists here as well. VT was brought in after these two, and he was quickly informed that this place, something about it wasn't quite right. Being he was the newcomer, VT figured they're just fucking with him, and he decided to chalk it up to a little teasing.

He ignored it and went about focusing on his designs. That is, until he showed up early one morning to find one of the cleaning crew hauling ass out of that building. She was visibly shaken, and before VT could even get a word out, asking if she was okay, she was gone. VT entered the lab, which, to his surprise, was empty. She was alone.

None of the equipment was on either. It was dead silent as to what in the holy hell happened, VT once again chalked it up to, well, maybe she saw a rat or even a wildcat. They are in the area. Well, VT shrugged and went about his day and went about his week, which, as you could probably guess, didn't go as planned. So what happened to VT?

Great question, my goose pimpled business. Goose. I'm going to read from the actual case study about VT's experiences as published in the April 1998 edition of the Journal of the Society for Psychical Research. You're going to have to shut the lights for me. We're getting spooky.

Here we go. Quote as time went on, VT noticed one or two other OD events. There was a feeling of depression, occasionally a cold shiver. On one occasion, a colleague sitting at the desk turned to say something to VT, thinking he was by his side. The colleague was surprised when VT was found to be at the other end of the room.

There was a growing level of discomfort, but the workers were all busy and paid little attention. That is, until VT was working on his own. One night, after everyone else had left. As he sat at the desk writing, he began to feel increasingly uncomfortable. He was sweating, but cold, and the feeling of depression was noticeable.

It felt like those wildcats were moving around, and the groans and creeks from what was now a deserted factory were spooky. But there was also something else. It was as though something was in the room with VT. There was no way to get into the lab without walking past the desk where VT was working. He looked around and even checked the gas bottles to be sure that there was not a leak in the room.

There were oxygen and carbon dioxide bottles, and occasionally the staff would work with anesthetic agents, all of which could cause all sorts of problems if handled inappropriately. All of these checked out. They were fine. So VT went to get a cup of coffee and returned to his desk. As he was writing, he became aware that he was being watched, and a figure slowly emerged to his left.

It was indistinct and on the periphery of his vision, but it moved as VT would expect a person to. The apparition was gray and made no sound. The hair was standing up on v t's neck, and there was a distinct chill in the room, as VT recalls. Quote, it would not be unreasonable to suggest I was terrified. End quote.

VT was unable to see any Detail and finally built up the courage to turn and face the thing. As he turned, the apparition faded and disappeared. There was absolutely no evidence to support what he had seen, so he decided he must be cracking and went home. End quote. Well, my equally terrified business goose, you're probably thinking, jesus, I am hoping he called out sick the next day, right?

Or quit. Did he quit? Because I would. And it wouldn't even have to be haunted. Well, get this.

Luckily, VT had the following day off, but he went in anyway. Yes, you heard me. For you see, VT had a little hobby. He was really into fencing. I know.

Yes, fencing. You heard me. Lost my place. But he happened to be in a competition. Unfortunately, a handle to one of his foil blades had detached, and it would only take five minutes to fix it.

If he went into the office and used one of the vises there to hold the blade in place as he reattached it. Okay, so five minutes. What do you think? What could possibly happen in five minutes? VT thought, not that much.

So he went in. The office was empty, thankfully, so Vt went back to work. He clamped the blade to the vise and walked to another spot to gather up whatever the fuck else he needed. When he returned, my friends, the blade was, quote, frantically vibrating up and down, end quote. It was as though someone had pulled the blade down and just let it go.

Are you screaming? Because that's the normal reaction, but VT doesn't seem to be the average person. Yes, he reported he was scared, but he was also curious. He thought, and I quote, quote, if the foil blade was being vibrated, it was receiving energy which must have been varying in intensity at a rate equal to the resonant frequency of the blade. End quote.

Okay, now that wouldn't be going on in my head at the time, but good for VT. This was the beginning of VT's investigation, and it focused on sound waves. Yes, my speechless business goose. VT theorized that maybe, just maybe, one of the many sounds from the many devices could be what's making the foil blade go frantic. So, being the proper researcher, he walked the blade still in its vise, through the workspace, and when you know it, at a certain point, it completely stopped moving.

It vibrated most at the center of the long corridor and stopped moving when at the end, doing some clickety clack calculations, which I have a screenshot of over at our social media stuff. So please come on by and take a look at the equation. I'm not going to read it. VT figured out that the blade was being impacted by a mere 19 hz sound wave. 19 level.

My friends, we cannot hear, but definitely affects us. According to a little agency called NASA, this level of infrasound can cause hyperventilation, panic attacks, and incite fear. Don't worry, it gets better. As noted in Kromer's 1994 edition of ergonomics, how to design for ease and efficiency sound between two and 20 head to feel, quote, general discomfort. And between 20 and 70 hz eyeballs can vibrate, which can affect your vision.

Okay, so. Son of a bitch. Tis no ghost we're dealing with, but the horrors of a poorly designed workplace. It's truly frightening. Now, VT asked the building's maintenance workers.

What in the holy hell could be humming at 19? Howdy. Wouldn't you know it. A new extraction ventilation fan was installed above the washing station. Right around the time all of this began, a few modifications to the fan were made, and slowly but surely, all of those symptoms faded away.

I think the real horror is that thanks to VT, no one can use the excuse of a poltergeist to call out God damn it. And thank you for listening rating subscribing telling your friends maybe not to say anything when you take a photo just to be safe. Or maybe to yell when you take a photo, just to be safe. I don't know. So just go with whatever your heart tells you, and tell them about VT, who almost went insane from working.

Don't go to work, everyone. And a boost size thanks to the folks over at airwave media, the podcast network to which WTI belongs. If you love this show, and you do, you'll love the other podcasts in this family. So please these and please stay interesting close.