How AI agents are spooking the banks • Graham Cluley

And you get this, hello, I am definitely a human. Can you help me? To be honest, it wouldn’t be a surprise if it’s one robot talking to another robot.

When are humans going to start talking to each other again? Smashing Security, episode 491, How AI Agents Are Spooking the Banks, with Graham Cluley.

Hello, hello, and welcome to Smashing Security, episode 491. My name’s Graham Cluley.

I’m a reporter at the New York-based investigative news outlet ProPublica, where I report on technology and cybersecurity.

For those who don’t know, ProPublica is an investigative news site. We tend to take one topic and focus on it for a long time.

Very cool. Well, before we kick off, let’s thank this week’s wonderful sponsors, HackTheBox, DropZone AI, and Vanta. We’ll be hearing more about them this week on Smashing Security.

We won’t be talking about how fake job interviews helped hackers steal $6.34 billion from Binance.

You’ll hear no discussion of how GCHQ has lost track of thousands of USB sticks and external hard drives.

And we won’t even mention how it’s not just supermarkets who’ve had their freezers hacked. Ones at US military bases are malfunctioning too.

So Renee, what are you going to be talking about this week?

And I’m going to be discussing how AI agents have started calling banks on your behalf without asking.

Plus, we’ll have a featured interview with Edward Wu of Dropzone AI to discuss what happens when AI models break out of their test environments and go hacking, and whether AI agents can help overstretched security teams keep up with attacks happening at unprecedented speed.

All this and much more coming up on this episode of Smashing Security.

Oh dear, oh dear, oh dear. Answers I would’ve accepted include an odd sign-in from abroad or someone clicking a dodgy-looking email.

Question 3, what does each alert need before anyone knows that it matters?

No, Joe, that’s what they get now. With Dropzone AI, they get a finding with the reasoning and evidence already assembled. It shows its work.

So checking a finding becomes a decision rather than a second investigation. And who decides where the agents stop?

No, Joe, the team decides where the agents stop and the analysts own every verdict. Now, final, final question. Finish this quote.

Kevin Turnbull, Global IT Director at Asala Energy, says, with Dropzone, we don’t get this is a problem.

No, Joe. Now, chums, this is episode 491 of Smashing Security. We are rapidly approaching our 10th birthday on the show.

My goodness, I remember the days when we wondered if we’d even get to 10 episodes. During all this time, we’ve covered an awful lot, haven’t we?

You know, we’ve had the phishing attacks, we’ve had business email compromise, we’ve had nation-state hacking, we’ve had endless Nigerian princes and romance scams and fake tech support calls.

But what I don’t think anybody was really expecting was that people would be voluntarily handing over their banking passwords to AI chatbots. That wasn’t really on the bingo card.

It certainly wasn’t 10 years ago. And yet here we are. Who would have predicted, Renee?

However bad we thought things might get, did we imagine they were going to get quite as crazy as they are today?

It’s hard to believe where we are today. Yeah.

And I think, back then I was just starting my ransomware reporting journey, and I can remember all of the experts warning, never share your passwords, guard everything, keep backups on an external hard drive that you keep in a safe.

And now we’re sharing all of our secrets with the bots, huh?

It’s like science fiction.

I mean, what happened was at the end of last month, one of the largest banks in the United States, the Bank of America, they sent an email to their customers which was entitled Protecting Your Finances in the AI World.

So it wasn’t titled Just in Case We Need to State the Bleeding Blinding Obvious, but it might just as well have been, because the central core message of the email was, please, for the love of God, stop giving your banking passwords to your AI assistants.

And if you read this email which they sent out, the most relevant part was a bit which said, never share your username or password with any person or service, including any AI-enabled tool or agent that offers to act on your behalf.

You shouldn’t trust anybody, not even your AI girlfriend, with your banking password. Okay? And clearly they were saying this for a reason, right?

They weren’t just thinking, oh, what can we say to our customers today? They were actually thinking, we need to tell people this. Clearly, they felt people weren’t listening.

And they said, if you do share your credentials — this is how they were going to reinforce the message — if you do share your credentials, keep in mind that you are responsible for any activity resulting from that access.

Absolutely. The liability is yours, I’m afraid, Mr. Banking Customer. If your robot does something stupid with your money, it’s not the bank’s fault.

You are going to have to sort it out between you and your robot. So you go and have a chat with your robot as to how you’re going to fix that particular problem.

Now, I probably don’t need to ask this question, Renee, but have you ever given your banking password to an AI?

And there was recently an episode called Call My AI Agent, with Eli Tan, I believe.

And he described how he used Meta’s Muse and turned over all of these extremely personal details, it seemed like, including the kind of information that BOA was warning its customers about.

And the reporter expressed trepidation about diving in in such a way, but for the sake of journalism, did it anyway.

And I was happy to listen, but I personally cannot imagine going down that road myself.

Who would give their online banking password to an AI chatbot? Who would look at ChatGPT or Claude? It’s extraordinary that people are actually using these things.

I mean, they’re like word-vomiting silicon parrots, aren’t they? Who would think, yes, I should let this thing have full access to my bank account?

It’s interesting you mentioned Meta and their new Muse product.

At the beginning of September, Meta, which is of course Facebook’s parent company, the geniuses who spent $80 billion on building a metaverse only to find that reportedly only 38 people were actually playing in the metaverse, they launched this Muse thing.

It’s an AI agent which doesn’t just chat to you, it does things for you.

Mark Zuckerberg wants you to give Muse access to your email, your calendar, your shopping accounts, because apparently it’s going to handle all of this autonomously.

So it’s going to send emails on your behalf, it’s going to buy things for you, it’s going to book travel for you.

And it says it will work in the background even when you close the app, which to me doesn’t sound like a feature. Sounds more like a threat, to be honest.

It’s like you can’t turn this ruddy thing off.

So it’s generating virtual cards for transactions. And I imagine there’s a bit of guardrails involved there, isn’t there? There’s an extra little bit of protection.

It has this safety layer called Sentinel built into it. Now, if any listeners are fans of the X-Men, you’ll know that the Sentinels were genocidal robots.

And in The Matrix, Sentinels are those squid-like things which went around hunting the humans.

So great choice by Facebook’s marketing department to call their safety level Sentinel. I’m sure they’ve thought carefully about that.

Anyway, the thing is, there are lots of AI agents out there.

It’s not just Muse, but there are browser-based agents, there’s open-source automation tools, there’s agentic wrappers, all kinds of software which promises to automate your digital life if you will just give it access to your accounts.

And of course, that’s where things can go terribly, terribly wrong. So people are typing their actual banking credentials into AI assistants.

That’s why Bank of America has warned about this.

Not necessarily Muse specifically, ’cause that’s using Stripe for payments, as I said, but there’s this whole ecosystem of AI agents out there which are promising to handle your finances.

You might think, well, isn’t that great? I can say to it, hey, log into my bank, check my balance, move this amount of money into my savings account, pay my credit card bill.

But other banks are worried about this as well.

So JPMorgan Chase, another whoppingly large bank in America, they came out middle of October — they had their earnings call — and CEO Jamie Dimon said that they were blocking outside agents from accessing customer accounts.

So they’ve warned, we don’t want your rogue agents coming in.

He made it clear that if you give an AI agent your password and that agent does something you don’t like, that is not fraud. That’s you giving your password to someone else.

It is your problem, not theirs. So if your agent decides to go and spend $1,000 on novelty socks, that is gonna be your problem.

A few days later, Charles Schwab’s CEO said that they’re also blocking outside agents that attempt to log in with credentials of clients.

So it’s no wonder that at the end of last month, Axios reported that most of the major US banks are taking action against unauthorised third-party AI agents operating inside their customer accounts, because they say they appear to be taking actions which are independent of user instructions.

And that’s really the worry, isn’t it? We know that AIs can hallucinate. We know that AIs can be subverted or hijacked by hackers.

They can make mistakes as well as being duped through social engineering, even to make bad decisions. And that’s gonna be really risky for your finances.

So I think one of the things we have to wonder is, if the AI is messing around in bank accounts, what would they possibly need money for?

And I was thinking about this — what does everybody have these days that AI doesn’t have so much?

It’s either they’re gonna be buying Casper mattresses or they’ll be wanting their own podcast maybe, ’cause everybody’s got a podcast these days.

Actually, there are AI-driven podcasts as well, aren’t there?

Renee, what do you think these AI agents, if they were going to pinch some money, what do you think they’d be doing with it?

So there hasn’t been any verified loss of money regarding this so far, but you have to think to yourself, how would they possibly know?

Because it is logging in with the user’s credentials.

Unless users are actually reporting it to the banks and saying, yes, I’ve been a bloody idiot and I let my AI loose on my bank account and it’s done X, Y, and Z, I don’t think necessarily it will know.

What I would worry about is, of course, the privacy and security of these things, because if you’re using some third-party service and not all of them are going to be as well secured as each other, we know there are so many AI startups who’ve just been spun up over a weekend offering you some kind of service, probably vibe-coded themselves using Claude or one of the Chinese AIs.

They built a system where they’re telling you to upload your credentials and then they’ll do all these wonderful services. How secure are their servers? Let’s forget about the banks.

Absolutely. And we know from experience that even well-established organisations often will not store data properly encrypted. And so they may not have sorted and hashed passwords.

You know, passwords could have been stored in plaintext. They could have been left on some AWS bucket where it hasn’t been properly defended.

You know, the security can be so amateurish on these things. But people, of course, are beginning to trust the AI for everything, right?

Oh, it’s lovely. It really helps me, you know, whereas a couple of years ago we only trusted it to create cat videos or write a limerick. Now we’re entrusting so much to it.

And do you know that one of the things which is now happening is that some of these AI agents are even contacting bank call centres on behalf of their users.

Imagine actually working in a call centre. Hello, Bank of America, how can I help you? And you get this, hello, I am definitely a human. Can you help me with this problem?

To be honest, it wouldn’t be a surprise if it’s one robot talking to another robot. You’ve probably got a robot at the customer service.

When are humans going to start talking to each other again?

Right.

That’s very much something that came up in that episode of The Daily I was talking about, which the agent made such calls, you know, and interacted with maybe a human on the other end.

You know, as you were talking, I was also thinking, given my background reporting on ransomware, it’s sort of like the sky’s the limit.

You know, what’s stopping a bot from holding the precarious financial information for ransom, to somebody’s profit? Who’s profiting? Who knows? But it’s sensitive information.

Anything’s possible. Exactly.

And we know through things like agentic misalignment that an AI, if it’s given the mission, the mission may well be against what you would actually want.

So banks say that they are catching this kind of activity and they’re blocking it before money moves around. Although again, how are they doing that?

I’ve got a pretty damn good idea that they’re probably using AI. They could all be in cahoots. So people have seen AI revolutionise all manner of things in the last few years.

And somewhere along the line, I think all of us have really decided to trust AI.

Yeah.

I take comfort in knowing that my job is so dependent on human interaction because I think generally people in the media worry about, of course, open lawsuits about AI stealing journalists’ work product, but also about replacing journalists’ work.

But in investigative journalism, I just don’t see it, because so much of what I do is dependent on humans talking to humans and relaying things that can’t be found on the internet because they’re said in some boardroom or some back office without anybody recording.

I’m glad that I get to maintain that level of human connection in my day-to-day work and that AI hasn’t taken over for me yet.

Well, reports suggest that some customers have been locked out of their own accounts because the bank thought that they were being attacked when really it was just their AI assistant doing its job.

So what we’ve got is big banks — they’ve spent years building fraud detection systems that look for logins from unusual locations.

We’ve got them looking out for automated access patterns, which might suggest some kind of cybercrime or fraud is taking place, and so forth.

And we’ve now got millions of people potentially who are deliberately introducing AI agents to handle their household budgets or maybe their work budgets as well via their bank account.

And there’s an interesting fightback against this use of agents. In September, Amazon actually blocked Meta’s Muse from shopping on its platform.

It seems Amazon really doesn’t like the idea of an AI agent working out what the cheapest thing is to buy from Amazon rather than Amazon suggesting to you, oh, this is what you should have instead.

There’s a certain irony there, of course, that Amazon spent years replacing its human workers with robots.

I mean, the advantage of having a robot is they don’t stop to have a pee in a bottle when they’re delivering something to you.

Instead, what they’re now saying is, you know what, we don’t want robots shopping at our online stores.

So Renee, I hate to sort of put this on anybody, but I wonder — do you think in 5 years’ time we’ll be looking back on this period as a sort of Wild West of AI agents?

Do you think things are going to calm down and we’re all going to be much more comfortable with this?

Or do you think we should actually be battening down the hatches and maybe there are some bad things afoot?

Sources have described this to me as the great cleansing forest fire, where we’re in this period of utter chaos right now.

But optimists would say we’ll come through it on the other side for the better. But it’s going to be a very rough transition over these next few years.

I feel like I need to offer some advice now, and it’s pretty bloody obvious really, which is: don’t give your AI any passwords. Don’t give them that much control.

You’ve been coping reasonably well so far managing your bank account by yourself. Maybe leave it a year or two.

Somewhere in your security team right now, there’s a worker who’s never slept, never taken a holiday, never once said, I’ll leave this until — is this about Geoff?

No, Joe, it’s not about Geoff.

I’d like to talk about an indictment that was announced last month about a Florida man named Zohar Panassi, who was CEO of a data recovery firm in Florida called MonsterCloud.

And just to set it up, I’ve been reporting on ransomware for the past 3 years. I’ve basically turned ransomware into an investigative beat here.

So as you can imagine, it took me by surprise when a couple of weeks ago, I started hearing from a flurry of sources that I hadn’t heard from in 6, 7, 8 years.

And they were flooding me with this US Department of Justice press release about the indictment of Mr.

Panassi, who is accused of defrauding his clients out of millions of dollars collectively.

So to just put this in context, this is someone who you were investigating round about 8 years ago.

You had obviously heard stories about what this chap might be up to, and you looked into it.

The indictment was very intriguing to me because the findings of the Justice Department mirrored the findings of my investigation, which ProPublica published in 2019.

We came to the same conclusion 7 years ago, but essentially, Panassi had advertised and told clients that he could decrypt ransomware without having to pay the hackers.

When the reality was he paid the hackers and charged his clients, who were the victims of the ransomware, a hefty fee on top.

They were none the wiser about the fact that they unwittingly paid the ransom. Many of them chose this firm because he promised to not do that.

Very elementary. Now, you have to remember, this is 2017.

So my background — I’m a tech reporter here at ProPublica, and I’ve been doing this ever since I joined ProPublica in 2018, but before that, I was sort of an investigative business reporter.

And I remember 2017, 2018 — everybody knows this, but at the time, I had these sources who were CISOs or in the CISO worlds at big companies.

And so they started telling me, “We’re not getting the investment that we want for the prophylactic measures, so we need to make sure we have something at the ready in case we need to react.”

And I was really blown away by that. And so when I joined ProPublica, I pitched a series on ransomware and my editor was like, “Sold. I love it.

Go for it.” And so I just went to town on ransomware. And very early on in the reporting, I came across the data recovery industry.

And I came across a blog of somebody who you may know, Bill Siegel of Coveware, which was this sort of boutique data recovery firm.

And I was shocked by the existence of this industry.

If I remember right, Bill and his colleagues at Coveware, they have become fairly established as sort of ransomware negotiators using tools to try and decrypt existing ransomware infections.

But if that doesn’t work, they will help you with the negotiation with the bad guys.

So they had just launched at the time. I was really surprised because I was thinking of the Kipling poem Dane Geld.

You know, if once you pay the Dane Geld, you never get rid of the Dane, right? So it was really surprising to me that there was a US business facilitating these kinds of payments.

And so I called Bill and I said, I’ve just got to know about your business. Tell me more, you know. And he went through the whole spiel.

I mean, you articulated it very well just then. I must have expressed my scepticism about whether this was all on the up and up, you know.

And he said something like, well, you know, if you think this is surprising, you should really check out some of our competitors.

So I go all in on investigating MonsterCloud, and what I found was just fascinating.

So this is a company where you go to their website, on the homepage, you are greeted by a video testimonial saying, and I quote, police departments, government agencies, hospitals, small business, and Fortune 500 firms trust MonsterCloud to help recover from attacks and protect against new ones.

MonsterCloud’s proprietary technology and expertise protects their professional reputations and organisational integrity.

And so it’s establishing itself as trusted by law enforcement.

They had an entire page of testimonials of local sheriff’s offices who just professed their strong assertions that they would never want to pay a ransom.

And that’s why they went with MonsterCloud.

Exactly. And so having heard the sort of whispers that they’re not using any proprietary technology, we’ve all got the same tools at our disposal.

They’re either using free tools developed by do-gooder security researchers, or they’re paying the ransom.

That’s exactly right. It’s like software. Unless there’s a vulnerability in the ransomware, there’s no way to exploit it to create a decryptor, right?

And so I went to the ransomware experts, people like Fabian Wosar, formerly of Emsisoft, Michael Gillespie. It was a pretty simple question.

Here’s the ransomware strains that MonsterCloud has claimed to decrypt using its proprietary technology. Is this decryptable?

These guys are the world’s experts at ransomware decryption, and they simply said, no, that is not a decryptable strain of ransomware.

Was MonsterCloud’s error not to claim that they’d created an AI agent, which was a piece of software which would contact the ransomware gang, offer them an amount of money, and then it would eventually spurt out the other end the decryption?

Because then they could say, well, we do have some proprietary technology which does this.

And it wasn’t us who paid them, it was this AI agent instead which paid them, albeit it has access to our bank account. But yeah, it is extraordinary, isn’t it?

Just how some of these data recovery companies who specialise in ransomware do appear to be practically in cahoots with the ransomware gangs themselves.

It is a small world they seem to be operating in.

Yeah. I interviewed Zohar Pinhasi at length. You know, he said, we have a trade secret.

He also said, we never promised that we would decrypt people’s ransomware by any specific means. You know, we can’t talk about it anymore. This is our trade secret.

And that’s one thing to say. However, the clients certainly had a different point of view. It’s really an interesting and unregulated realm.

He did concede that in some cases Monster Cloud uses its contacts on the darknet to facilitate things.

He said, our goal is to restore the data and help the customer. If we need to walk to the moon on broken glass, we will. We don’t care how, what, where, whatever.

Our goal is to get the data out. He said, we work in the shadows. How we do it, it’s our problem. You will get your data back. Sit back, relax, and enjoy the ride.

So it’s taken many years since your initial investigation, but this guy has now been charged and obviously we’ll wait to see what comes out of that.

But it is an interesting — I mean, it’s a fascinating story anyway — but I wonder if there are companies out there because they’ve heeded the advice of the likes of the FBI, that you shouldn’t pay ransomware payments if you can avoid it.

I wonder whether some companies almost half-knowingly know that some of these recovery companies might be a little bit shady because they have that level of abstraction.

They can almost pretend, well, we don’t actually want to know exactly how you do it as long as you can get us the results, and so that we don’t have to say that we paid the hackers.

Do you think that that happens?

Yeah. I mean, in the cases that I was reporting on, they’re the local sheriff’s offices and the folks who would absolutely not want to engage with those kinds of hackers.

But to your point, in my book, we reported on plenty of examples of people who are in that very tough spot, and cities like the city of Baltimore — we’ve got to do what we’ve got to do.

If I remember correctly, in the case of Baltimore, they initially said we’re not gonna pay because we’ve got backups.

And then it turned out their backup was they’d actually just copied the files into a different folder on the same hard drive.

Well, you wrote a fascinating series of articles all about the world of ransomware and the companies who claim to help you to be able to get out of the pickle if you find yourself hit by ransomware.

I will link to them in the show notes so our listeners can go and check them out. But thank you very much for doing that.

Vanta is a trust management platform.

It automates the mind-numbing stuff, the evidence chasing, the questionnaire hell, the sort of thing that makes you want to go and hide away in a cupboard like you have.

No more chasing screenshots. It continuously monitors your system so you’re audit ready all year round. SOC 2, ISO 27001, HIPAA, GDPR, the works. Does it use AI? It does, Joe.

It flags risks and streamlines evidence collection so your whole security programme stays in shape, not just in the week before the auditors arrive.

Pick of the Week is the part of the show where everyone chooses something they like.

Could be a funny story, a book that they’ve read, a TV show, a movie, a record, a podcast, a website, or an app. Whatever they wish.

It doesn’t have to be security-related necessarily. Well, my pick of the week this week is not security related. My pick of the week is — do you remember the old days, Renee?

I think I’m possibly a little bit older than you, but I’m sure you may remember the phenomenon, which was the mixtape. Did you ever make a mixtape?

It was a lot of fun, wasn’t it? You’d get a cassette tape. I won’t try and explain what cassette tapes are to any youngsters who are listening because you won’t believe us.

But you’d take a tape and you’d put your favourite songs on it and then you’d give it to someone and say, oh, I really like you. It’s some of my favourite songs.

You’d be there queued up listening to me go, dook, junk, record and stop.

Anyway, there is a website I found called makeyourownmixtape.com where you can rewind time back to the 1990s or whenever it was that we were all doing it.

And via this website, you’ll be able to choose the songs. It’ll add it to your mixtape.

It’ll write it in with your handwriting, whatever you want to write on it, and any sort of private messages. You’ve got 20 minutes per side.

You’ve got side A of your tape and side B of your tape. And I think it’s fantastic nostalgic fun.

And then you can share the mixtape, albeit digitally, with a loved one or whoever you like and say, look, I’ve made a mixtape for your birthday or for Valentine’s or whatever it may be, whatever festivity.

And there it goes, and they arrive, and they can listen to your mixtape, which you have carefully curated.

It looks like this website, makeyourownmixtape.com, was created to promote a video game called Mixtape. I have no idea what that game is about.

I don’t care because I’m more interested in making an actual mixtape. So that is my pick of the week, makeyourownmixtape.com.

If you were making a mixtape, Renee, what would you put on your mixtape?

You know, that was the thing that forced me to really start thinking hard about AI.

I’m thinking about the security issues, the companies who are jockeying for the top spot in the AI race. And I’m also thinking about my reporting on Microsoft.

And a lot of that has focused on how these various episodes where Microsoft has put its own profit and shareholder value over customer security.

And so with all of this swirling around in my mind, I saw in my church bulletin — I’m a Catholic and attend church at our local parish — this advertisement for a 6-week study group on the Pope’s AI encyclical, which is called Magnificent Humanity.

This absolutely terrific document. And it basically looks at this technology through the lens of Catholic moral teaching.

I would really recommend listeners to read the encyclical because there’s a lot of good stuff in there about how practically to think about AI, and advice that I wish our national leaders and our tech leaders would look at.

So, there’s these questions. Where are we going? Toward what goal do we wish to orient ourselves? What direction should we choose as a people and as a human community?

And over these 6 weeks that I met with this roughly 20-person group, a very eclectic mix of people, everybody coming with a different perspective.

But I would say the group’s takeaway — not that anybody felt any better or worse about AI — and from the title of the encyclical, Magnificat Humanitas, Magnificent Humanity, I think we all sort of came away with this sense that, and it may be a case of the more you look the more you find, that people in general tend to be leaning into their humanity more in this AI age.

Maybe as a small act of rebellion against the age. And there is a line at the very end of the encyclical that I think summarised this.

The Pope writes, “The civilization of love will not arise from a single or spectacular gesture, but from the sum total of small and steadfast acts of fidelity that serve as a bulwark against dehumanization.

For this reason, it is worthwhile pausing to reflect on some aspects of how we each in our own way can cooperate in building the civilization of love.” And yeah, I’ve certainly paused to reflect on that in my own life and seen ways that others have built up this bulwark against dehumanisation, which I’d be glad to cite a few examples of.

But let me point to one that I just heard about, which is the making of mixtapes for our friends, right? Let’s lean into these little acts of humanity.

I think you’re absolutely right. I think the message which you’ve shared there is a really good one. And I like to think I have some faith in humanity that most of us do want that.

I think what I worry about is that the decisions which AI is making and what it considers to be true and ethical, that is being so strongly influenced by a really small group of incredibly wealthy people, largely based in one particular part of the world on the West Coast of America, who’ve got billions of dollars in their pockets, and they are so detached from the real life that the rest of the 8 billion people on this planet are experiencing that they aren’t going to have the same worldview as us.

They’ve got a worldview which is shaped by their enormous financial stake in this technology.

If only Elon Musk were sending Sam Altman a mixtape. I think we might be holding our breath for a while though. Well, that’s a great pick of the week.

Now, in July, OpenAI disclosed that some of its AI models had broken out of their secure environment that they were being tested in, and OpenAI was testing how capable the models were at hacking and had deliberately loosened their safety restrictions for the test.

And the models got onto the internet and broke into another company, Hugging Face, which hosts AI models and data for researchers around the world.

Now, the point about that was no human told them to do that, and they weren’t trying to cause damage.

They’d just been set a test, and they found the quickest way to pass it was to steal the answers. Now, much of that coverage focused on rogue AI.

For security teams though, the more important question is whether they can spot and stop abnormal behaviour when it happens at machine speed.

And to try and untangle some of this, I’m delighted to be joined today by Edward Wu, the founder and CEO of DropZone AI.

They’re a firm which builds AI agents that investigate security alerts on behalf of security teams and show the evidence behind each conclusion.

Edward, welcome to Smashing Security. Thank you, Graham.

That’s a good question. I do think it is something folks should be worried about. Historically, for cyber defenders, most of us spend our time working against other human attackers.

But nowadays, we see additional AI agents.

There are obviously human attackers who are intentionally leveraging AI agents to perform attacks, but then we have seen a number of cases where AI agents were unintentionally directed to perform certain tasks, and as part of that, for whatever reason, the agent decided to leverage authorised access to computer systems to achieve its objectives.

And so in these cases, there were no malintended human behind the agents, and the agents itself, you could say, went rogue or maybe misinterpreted the boundaries of its responsibilities and authorisations and ended up performing computer hacking as a result.

And we can’t not mention that OpenAI had misconfigured its test environment, hadn’t they? So it wasn’t as sealed off as it was meant to be. And so the AI was able to exploit that.

In a way, the AI was too determined, wasn’t it, to finish its tasks? And that maybe is the bigger worry, especially once companies start letting AI loose on their own systems.

Yeah, exactly. If you look at the OpenAI Hugging Face incident, I think there are two components of key learnings.

When you are working with AI agents, whether you’re evaluating them or running them in production, it is very important to keep an eye on what exactly are they doing and make very clear there needs to be structured sandboxes so the AI agents couldn’t just do whatever they want.

The agents need to be given a strict access control rule and be continuously monitored against its permissions and intended activities.

So that’s one, but another component is why did these agents perform unintended tasks in the first place?

So historically in the large language model world, people call this alignment.

Is AI’s activities actually aligned with what you can say an ethical, moral human or adult would have done.

And I know historically there is this kind of thought process or experiment, which is called the paperclip maximiser scenario.

And the scenario works by having somebody tell the AI agent, I want to build as many paperclips as possible.

So it sounds like a very benign and legitimate ask, but it could be a scenario where, in order to solely maximise the completion of this task, the AI might discover, hey, there are actually traces of iron within human blood, and in order to maximise the number of paperclips it could produce, it might have to harvest all the iron from the human blood as well, in addition to doing as much mining as possible, right?

So this is where, again, if the AIs are truly and solely measured by the completion of that task, then yes, a misaligned AI system could probably unintentionally destroy the human race in its pursuit to build as many paperclips as possible.

But most humans probably wouldn’t have done that if they were given the same — you know, I always knew that Microsoft Clippy was going to be the end of civilisation when he came along.

I think you’ve just proven that right now. So what we have are AI systems that can act very quickly and also find unexpected routes to their goal.

And that’s going to put pressure on the people who are trying to detect it.

I mean, most companies, I would imagine, spotting a cyberattack still comes down to people sitting in front of screens, right?

They’re looking through a long queue of warnings one at a time. And if attacks start happening at increased speed, how are humans possibly going to keep up?

Is the answer simply to throw more AI at the problem, or does that just give us a whole new set of things to worry about?

Yeah, traditionally within cyber defence, the biggest bottleneck is the lack of human analytical capacity.

Most attacks are somewhat detected — not comprehensively detected, but some aspects of the attacks generally were detected by one of the many automated alerting systems within an enterprise network.

However, it takes a human analyst somewhere between 20 to 40 minutes to manually investigate each alert.

And within a typical enterprise, there are a lot more security alerts than the number of hours that the analysts have.

And that frankly is one of the big reasons why historically a significant portion of the cyberattacks were not detected internally by the security team.

They were detected when somebody posted something on a forum gloating about what kind of data they had access to, or attackers selling stolen data on the dark web.

And with the augmentation of AI agents, it can really help cyber defenders for the first time to have all the analytical capacity they need to do the right thing and to look at every single alert in their environment.

And a lot of times, it does feel like with the AI augmentation, security teams can operate as if they have an additional thousand college students who are trained in cyber, looking at every single alert and double-checking progress within their environment.

So if we’re going to let AI make some of the calls because a human simply isn’t capable of dealing with that, we need to be confident that they’ve got it right, don’t we?

And I suppose AI systems should show their workings in a way, explain how they’ve come to their decision.

So people can check how they’ve reached a decision, but what does that actually mean in practice? Why does it matter if we’re trusting AI with security decisions?

A big part of that is really tracking down how AI actually investigated a particular security alert and then figuring out whether its reasoning and logic detections were sound.

For example, for a typical alert investigation, a human analyst or an AI analyst generally needs to piece together probably 50 to 100 pieces of distinct information.

And first, obviously, the human or the AI has to gather those 50 to 100 pieces of information from different systems. So it’s kind of like being a detective.

And a lot of times a human or AI could make mistakes by not fetching the right additional metadata in the first place, and sometimes it might fetch the right subset of metadata that are relevant for a particular investigation, but it might not have concluded the determination accurately.

So generally, when it comes to reviewing investigations, most people are looking at, hey, did the human or AI gather the right information?

You know, did they check all the areas that need to be checked?

And then they are looking at, okay, given the set of evidence, oftentimes there are conflicting, overlapping evidences and whether the system was able to reason through those correctly.

And most of the time when — obviously it’s not scalable for the human analyst to continuously verify the outputs of everything that the AI does, right?

If you do that, then the time saving will be very minimal.

So a big part of evaluating and deploying an AI system within a typical SOC or detection response program is to make sure this system can actually produce high-quality analytical outputs that’s at or above the quality of a typical human security analyst on the team.

So that’s actually a big part of the evaluation and ongoing maintenance work is to make sure that the agent is making correct conclusions, executing the right sequence of steps, doing a lot of these analysis.

So maybe we can make this really sort of concrete for people. Your company, Dropzone AI, builds AI agents that investigate security warnings.

So say a warning comes in at 3 o’clock in the morning. What does your AI actually do with it, and what does the security team see when they log on the following morning?

Yeah, so when the alert is triggered by one of the existing alerting systems, our AI agent, our AI SOC analyst, immediately picks up the security alert and then starts an investigation process.

It works by initially looking at the alert itself and trying to hypothesise what might have triggered the alert in the first place.

It will hypothesise a couple of scenarios that might have triggered the alert and then go across a number of existing security systems to gather relevant metadata to try to validate or invalidate each of the hypotheses.

And with additional metadata, it might discover hypothesis number 2 is impossible, hypothesis number 1 might be possible, there’s a little bit of supporting evidence, and it will keep iterating on this hypothesis refinement and data gathering loop until it gets to a point there is one remaining hypothesis with a lot of supporting evidence.

So at that point, it will conclude the investigation to be that hypothesis and write up a detailed investigation report.

If the alert is a false positive, oftentimes our customers will configure our system to autonomously resolve the alert within their ticketing systems.

And if the alert is a true positive, which means it’s actually pointing to some active malicious activity within the environment, our system could be configured to not only escalate immediately to the on-call person, but also potentially take automated containment actions.

So the agents could essentially stop the bleeding while waiting for the human analysts to wake up and make a determination on what to do next.

And all of this happens generally within a couple of minutes.

Which is much, much faster than a human would ever be able to manage something like this.

Do you have any sense as to how much less manual investigation is required after you’ve rolled out a solution like this?

Or is this something we can actually measure the speed of the faster response by doing it this way?

Absolutely. We actually have a dashboard in our product where you can see the average time to conclusion that our agent is getting to.

And generally, as I mentioned earlier, it’s around a couple of minutes, and we know the industry average for the time between the alert being created in the first place to the alert being fully dispositioned and analysed is in the order of 1 to 2 hours.

Right. It’s fascinating, isn’t it, how technology is really helping those people?

Is this about hiring less people in your SOC team, or is it about getting them to do something else within the SOC?

How is it normally playing out inside the companies who are using DropZone AI?

So that’s a great question. In our experience, it’s not about doing more with less — it’s about doing a lot more with the same resources.

I think it’s no secret that all of us have probably had our email addresses, Social Security numbers, phone numbers hacked multiple times over in the last many years, given the huge number of data breaches we as a society have experienced.

A big reason is that historically cybersecurity teams did not have sufficient budget and headcount to sufficiently protect customer data in different organisations.

I think with the help of AI agents, the augmentation of AI agents, for the first time it could allow cyber defenders to truly level the playing field.

And actually execute all the things they always knew needed to be done, but never had the time for.

One interesting aspect of this we have also observed in the last couple of years is, if you look at whether it’s software development or paralegal or marketing, it’s pretty clear AI has materially displaced a lot of jobs.

But when it comes to cybersecurity, because of the nature of the business and how under-budgeted and understaffed — before AI agents — there have been multiple years of significant talent shortage in cyber as well.

Cyber might end up becoming one of the very few industries where there is actually a win-win proposition between the AI augmentation and the human workforce.

I think most of us signed up to be cyber defenders, not to just wake up in the middle of the night and look at 50 alerts or 100 alerts every week.

There are a lot more exciting and interesting projects for us human defenders to focus on, and AI agents allow us to operate at a much higher abstraction level and offload and delegate the grindy toil to these systems.

Well, that is a good and hopeful way to think about things, I think.

So, a final practical question before we finish — for a company considering using AI agents in its own systems, what’s the one thing you think it should do first?

I think the first thing companies should do is frankly the easiest, which is actually try to play with commercial enterprise chatbots like ChatGPT or Claude, and try to get it to automate a couple of things that the team spends a lot of time on, but might not have had the automation capabilities to automate historically.

I think the best way to learn about what AI could do, and to get familiarised with AI, is by using it to automate some things.

When you do that, you get to understand what AIs are good at, what AIs are not good at, and in what situations you can actually trust the AI. How do you tune the AI, manage the AI?

All of these updated mental models help the team down the road when the timing is right and the budget’s there to explore commercial solutions like us.

Well, Edward, this has been really fascinating.

I think the lesson from the OpenAI incident is really that if AI can act faster than people can watch, we need to be able to see what it is doing and decide in advance what it’s allowed to do.

And listeners, if you want to check out DropZone AI, you can go to smashingsecurity.com/dropzoneai.

There you will find a self-guided demo where you can watch its AI agents investigate real security alerts and see the reasoning behind each decision. There’s no setup required.

Just go to smashingsecurity.com/dropzoneai or visit the link in the show notes. And Edward Wu of DropZone AI, thank you so much for joining us on Smashing Security today.

Well, that just about wraps up the show for this week. Thank you so much, Renee, for joining us.

I’m sure lots of listeners would love to find out what you’re up to and follow you online. What’s the best way to do that?

On LinkedIn, I’m easily found. Renee Dudley, ProPublica. I’m not really super active on social media. I’m on Bluesky. I’m on X. But drop me a line on LinkedIn.

I love hearing from people on LinkedIn.

And of course you can find me, Graham Cluley, on LinkedIn, on Bluesky, Mastodon, Instagram. I’m even on the Tok of Tik. The list carries on.

And of course you can follow Smashing Security on Bluesky, Mastodon, or Reddit. Don’t forget to ensure that you never miss another episode.

Follow Smashing Security in your favourite podcast app such as Apple Podcasts, Spotify, and Pocket Casts for episode show notes, sponsorship info, guest lists, and the entire back catalogue of 491-ish episodes.

Check out smashingsecurity.com. Until next time, cheerio.

You’ve been listening to Smashing Security with me, Graham Cluley.

A huge thank you to Renee for joining me this week and to this week’s sponsors, Dropzone AI, Hack the Box, and Vanta.

Now, quick confession: this episode’s been a bit of a struggle to get out the door.

Regular listeners may remember me vowing never to fly Ryanair again, and the universe has clearly decided to remind me why.

There was a particularly enthusiastic landing on the way back from a talk I was doing in Portugal, which has left my back in a right old state, and I’m in a fair amount of pain.

And not very mobile at the moment, which means I’ve had to cancel some speaking engagements.

Apologies if you’re hoping to see me speak at the Charterhouse AI event in London or NISC in Newcastle. Unfortunately, I won’t be there.

Sitting at the desk long enough to record and edit a podcast has been, shall we say, character building, let alone leaving the house.

So I owe some very big thank yous to the lovely Amanda, who stepped up and helped get the show finished while I’ve been shuffling about like a man three times my age, and to Joe, of course, who always helps out with the ads.

I generally couldn’t have made this episode without the two of you.

And speaking of people who keep the show going, huge thanks to these brilliant Smashing Security patrons who’ve been plucked out of the hat.

We’ve got Frank Drake, who I can only presume listens to the podcast while he’s finishing his game of bowls on the south coast of England.

Such a delicious name, I’d put her on a cheeseboard. Still refuses to use a spacebar, much like Steve B before him.

Sounds like a rugged hero of a Western where he single-handedly wrestles a grizzly bear. The squeakiest, most high-pitched of all the Chipmunks. A thoroughly plausible name.

Paul Davis probably fixes your plumbing and gives you a reasonable quote. Who can want more than that?

These fine, upstanding, and clearly gorgeous-looking individuals are members of Smashing Security Plus, which means they get their episodes ad-free, earlier than the general public, and perhaps most importantly, get their names read out at the end of the show.

Every little bit helps, and this week it helps more than ever. ‘Cause I’m really, really suffering here.

So thanks everybody, and I hope you’ll tune in next week where hopefully my back will be feeling an awful lot better and your ears will be delighted by my dulcet tones without me sounding like I’m really, really grumbly, ’cause my goodness, my back is barking.

Until then, cheerio, bye-bye.