r/technology • • 21h ago

Privacy Researcher plants 2m canaries into the AI supply chain. A privacy disaster surfaced.

https://arstechnica.com/security/2026/11/a-leaky-ai-supply-chain-is-exposing-secret-user-data-at-an-alarming-rate/
1.5k Upvotes

105 comments sorted by

216

u/vessel_for_the_soul 21h ago

Don't talk to the confession box

93

u/Outrageous_Reach_695 20h ago

The Computer is your friend, Citizen.

20

u/thisischemistry 17h ago

Stay alert! Trust no one! Keep your laser handy!

9

u/Stilgaar 14h ago

Report commies

6

u/thisischemistry 14h ago

Reporting commies requires Form 201-Alpha-54-B.

Form 201-Alpha-54-B is currently out of stock. Report yourself for running out of Form 201-Alpha-54-B, using Form 102-Zeta-45-D.

Form 102-Zeta-45-D is currently out of stock. Report…

12

u/Ishmael128 14h ago

"Be a Government Informer. Betray Your Family & Friends. Fabulous Prizes to be Won"

5

u/glacialthinker 12h ago

I hate that all the games and shows I loved in the 80's are relevant in all the wrong ways today.

828

u/Sybertron 20h ago

I was recently working for a fortune 500 company and I had an AI chat open to talk through something I was working on asking it really general questions

It came back with something like "wow since you're at company x this idea would really fit in with the new strategy z they've been discussing" 

I had never said what company I was at, I was just on my phone. Strategy z had to have been something discussed only in a very private and secretive upper level meeting I certainly was not privy to. 

My guess is someone in that meeting was using the LLM to take notes or ask questions to and it just readily turned around and let someone know the Intel if asked the right question. Except I wasn't even trying to ask that question 

159

u/MysteriousAtmosphere 15h ago

Were you using copilot? Copilot has access to Microsoft graph which maps out all your email and SharePoint you have access too. My company found out the hard way that our SharePoint was set to default allow everyone at the comoabt to acess. meaning that copilot would look at all SharePoint including sensitive ones if the permissions wasn't changed.

This was not okay but not a huge problem until copilot came along

52

u/serrated_edge321 11h ago

2026 is like the biggest year of shakeup randomness. 😅

It's so crazy how everyone (applications, bosses, everyone) seems to be pushing AI so hard. It's like they all are getting individually paid for forcing the cool aid down our throats.

Meanwhile, I fight for hours with the idiotic chatbot to get reasonable results.

17

u/SIGMA920 10h ago • Bravo

They're the ones who put money into it or put some stake on it. If you just blew billions on an AI push that just crippled the company you work for in a sane world you'd be out immediately or at least seeing a lawsuit for your product doing so.

Sunk cost fallacy is one hell of a drug.

6

u/serrated_edge321 10h ago edited 10h ago

My company only has like 100 employees. I don't even have any tokens allocated to me. Their fervor for pushing AI is really just obnoxious.

So every now and then, when someone like my boss is complaining about something annoying he's gotta do... "Well, you can try putting chatGPT on that!"

I wonder if they've caught on yet that I'm just saying this as a joke because of how ridiculously hard they push AI...so far they just give this "Yeah, right... It's not going to work for that" type of look each time.

Maybe 2027 will be the year AI simultaneously becomes truly good-enough... And also f's up everything we take for granted in life. (I just took screenshots and made copies of my latest bank statements, just in case!)

0

u/SIGMA920 10h ago

It'll never be good enough to replace people, the money train will eventually run out through. Until then they're bagholders.

2

u/Corpomancer 3h ago

It's like they all are getting individually paid for forcing the cool aid down our throats

That's the neat part, they actually pay overpriced self-sabotage consultants to force it down their throats.

If only CEOs had an organization full of industry experts, with their company's best intentions at heart, at their disposal instead.

-8

u/smokedfishfriday 9h ago

Are you using a computer from 2022? How are you having so much trouble with the AI?

14

u/obxnc 11h ago

Our corporate copilot was supposed to be set up knowing enough about each end-user (company, title/job function, etc). I tried prompting it to write some sales emails for me and three different times it said I worked for one of our competitors instead of the company I actually work at. That's why I just stick to writing my own emails for the most part at this point.

4

u/gastropod-724 7h ago

Copilot serving me alarming documents from my email or sharepoint has not been confidence inspiring.

2

u/Sybertron 11h ago

Gemini iirc

2

u/Ok-Refrigerator 5h ago

My work Copilot also uses my Teams meeting transcripts. Which feels so stalker-y

1

u/Thoraxe474 10h ago

How do you check that?

284

u/ShanghaiBebop 19h ago

Your work account identifier is typically part of the system prompt identifying who you are. 

60

u/U3011 12h ago

That and IP blocks. If the company you work for has their own, publicly known IP CIDR blocks, it can figure out whomever you work for, too. Certain companies that shall remain nameless for the sake of clarity hold different blocks for different office locations, too.

All of these LLMs can see your IP address when you are in an active session. Even detecting network changes, such as going from Wi-Fi to mobile internet and back again if on a mobile device capable of getting tower signal or 4G/5G.

-2

u/Khaos1125 10h ago

Sure, but why on earth would you include the IP address in your system prompt? Not saying it couldn’t be done, but I can’t imagine choosing to waste tokens that way when building systems myself

10

u/Universe_Nut 8h ago

Data is everything. Tracking is everything. AI is part of the panopticons we keep in our pockets

44

u/Deep_Satisfaction556 14h ago

Soooo many ELT add bots and AI tools into meetings to capture action items and notes. No one has a clue where that data goes, but they know every word said, and by whom, and all is recorded for prosperity.

34

u/Ren_Kaos 12h ago

Posterity i think you mean? Tho prosperity is hilariously fitting.

8

u/Deep_Satisfaction556 8h ago

I need a typing bot apparently. 

1

u/Sybertron 13h ago

Yes exactly 

29

u/Buckets-O-Yarr 17h ago

Was "Strategy Z" even real or just made up by the AI? If you weren't privy to it, how do you know it wasn't just hallucinating that part?

64

u/LC8888 19h ago

Were you using a work account or your personal one to access the chat? From my understanding llms don't really learn after they are trained and released so I'm not sure how that could happen.  

62

u/A_Pointy_Rock 19h ago

They can access internal references, depending on the setup and use that as part of their outputs.

I asked one to give me a summary of a company's pros and cons the other day to sense check a list I had made. It found my list and parroted back the reasons I had given. Self-affirmation ftw!

9

u/LC8888 19h ago

Definitely and that makes sense, but if he isn't on his work account which shares context I don't know how the scenario could happen

8

u/A_Pointy_Rock 19h ago

It could also be a hallucination. It doesn't sound like OP knows for sure that strategy z is a real thing.

11

u/Fit-Produce420 15h ago

The model weights are fixed.

If it knew something past the training and cutoff it came from your own context cache or shared company account.

1

u/Sybertron 10h ago

It was just the public model, gemini I believe. Like if you just goto Gemini.google.com.

3

u/Fit-Produce420 7h ago

But were you logged in to a managed account owned by the business? Otherwise, no, it doesn't know proprietary info unless it was leaked to the model before training AND used in the training process. Doubtful they are training gemini on YOUR business plans, usually they train models on generalized data, or they are just memorizing facts which isn't ideal.

-1

u/Sybertron 5h ago

Nooo just mobile Internet from phone 

2

u/Fit-Produce420 4h ago

That's just not how it works, unless you have a group managed work phone.

-1

u/Sybertron 4h ago

You can't log into Gemini from a phone? 

2

u/Fit-Produce420 3h ago

That wasn't the point.

The point is that current models used fixed weights.

It simply doesn't know proprietary information about your company EVEN IF someone tells it, unless a new model is trained using that information. If someone tells model 1.0 a secret, it doesn't propagate through every instance of model 1.0 used by other organizations.

If model 2.0 is trained on that information for some reason then it would be in the data at any organizations using model 2.0. But things you say to model 2.0 still won't be available to others until they train model 3.0 on model 2.0's data.

6

u/serrated_edge321 11h ago

Wow, were you not even logged in to a company account for the chat bot? That's not terribly surprising tbh, given the way that C-level people operate... But it's bad for the company!

5

u/CrazedCreator 10h ago

This! I ended up in someone OneNote because I was looking for a policy.

1

u/serrated_edge321 10h ago

Omg... Which AI tool were you using? Copilot?

I need to protect my OneNote. 😱 Though I don't think anyone is using Copilot at my company...

1

u/CrazedCreator 9h ago

Yes copilot and if you have it on a open SharePoint then that's really when it's vulnerable. If you have it only on your personal one drive you're probably fine.

1

u/serrated_edge321 8h ago

I've never used Copilot before...

You would need to actually activate it / perhaps even pay separately for it, right?

1

u/Sybertron 10h ago

I was absolutely not logged into a corpo account and I believe my chat was mostly me asking about other jobs since I knew my role was ending.

1

u/Head_of_Lettuce 1h ago

Respectfully, I think you're leaving out some details here. There is quite literally no way for the LLM to have access to your data unless you've given it your data — whether you realize you have or not. LLM's are amazing tools and in many ways, dangerous. But they aren't magic.

If you were not on a corporate device or logged into a corporate account, this story is nonsensical.

1

u/Sybertron 1h ago

I am but I'm also some random engineer contractor that comes on site... 

1

u/Head_of_Lettuce 1h ago

My point is that what you're claiming is literally impossible, as described.

11

u/madsci 12h ago

I had Claude take a look at a competitor's site to see what framework they were using for something, and it came back and told me about an unreleased, unannounced upcoming product that it worked out a lot of details of by looking at sections that hadn't been enabled yet. I hadn't even asked it to look into anything like that.

7

u/Sybertron 10h ago

Ya my new company we just showed up at a vendor that we havent even signed up for yet, and they had claude do a whole automation line for the product we have absolutely secret and is not nearly to release yet.

I'm like I like your gusto, but yaaaa that would violate NDA for me. But hey I'm just the way down the line contract engineer.

3

u/He_is_Spartacus 17h ago

I've recently been pressured in to using Claude at work. One of the early questions I asked it was "do you talk to other Claudes within our company?".

It said no, it categorically didn't. Is there a different way to ask the question I wonder?

10

u/TeutonJon78 16h ago

Do you share data/info/history?

Probably some weird dodge about it being an AI and not "talking'.

4

u/He_is_Spartacus 16h ago

I share absolutely nothing that I don't need to. I've been privacy-focused since loyalty cards were released in the early naughties, refusing to use them as what I spent my money on would give a shit load away about who I was and what I did.

That's an interesting take though, I'll keep probing it

5

u/TeutonJon78 16h ago

Ha, I meant that is maybe the question to ask instead of if it's talking.

1

u/He_is_Spartacus 9h ago

Yup, I got that friend, which is why I said I'll keep probing it i.e asking it in a different manner

1

u/dalgeek 1h ago

I caught one of our PMs using a public LLM to summarize internal and customer meetings, even though we have a private AI assistant built into our meeting platform. They were making extra work for themselves AND leaking confidential data to the public.

1

u/beautybalancesheet 38m ago

... and yet, after spending lots of time setting up knowledge base, readily available for our company's Claude, it still didn't know who is who in our team. :D It started arguing about an email signature being "misleading". Took three "why's" before it acknowledged the mistake.

0

u/Extension-Rabbit3654 13h ago

Lmao, thats insane, no one was sanitizing their inputs and putting proprietary company info into a public model?!

We're not even fortune 500 and we have our own internal, locally hosted models for stuff like this

1

u/Sybertron 10h ago

Taking notes is hard I'll just have this AI do it for me is the rationale explanation.

I get it, but also dont do that morons.

0

u/[deleted] 19h ago

[deleted] — view deleted comment

1

u/Sybertron 19h ago

I was on mobile at the time so it could have guessed by location, but even then we had all sorts of vendors and visitors that come in and should not be knowing things and I'm gonna go ahead and error in the side of "they didn't really set the LLM up with the proper information security walls" 

1

u/Ishmael128 14h ago

Did you report it?

2

u/Sybertron 13h ago

Nah I'm a lowly contractor. Far more likely to come back at me as something I did

163

u/Interesting-Baa 21h ago

If I'm reading this and the Dragos article correctly, this experiment was testing the AI router services like OpenRouter and Microsoft Foundry, not the LLMs like Claude or CoPilot directly? Does it include hosting providers too?

125

u/webternet 20h ago

You mean you didnt comment purely based on the headline? Did you know that was illegal here?

25

u/Interesting-Baa 20h ago

I'm an optimist :) Well, sort of - if I am reading it correctly, it could still be a security company laundering advice from "upstream providers" like OpenAI and Anthropic. Don't buy from those cheap and nasty resellers! Just buy from our cult leaders instead!

4

u/Conscious-Tutor3861 10h ago

The security researchers actually recommend on-premises models or, failing that, a "single privacy policy" provider.

2

u/Interesting-Baa 8h ago

Yeah, but they do go into detail about how expensive on-premises models are, which is pretty discouraging. Theres not many businesses or individuals who could afford it, compared to the number of people who use the router services because they want the most cost-effective option they can get. Dunno, maybe I'm just over-doing the scepticism.

2

u/Conscious-Tutor3861 8h ago

They do mention on-premises models as being the more expensive option, and therefore suggest the "since privacy policy" models as a cheaper (but less secure) option.

4

u/SeanBlader 19h ago

Heh, I admit I often don't bother reading, but Ars is fantastic and that was relevant to my interests, so I'm one of the few who did read it.

10

u/farmerau 19h ago

Having read both articles, I’m not sure I see a reference to Microsoft in either? In the first, I see lots of references to AWS? Or was this just an example from yourself?

6

u/jaktonik 14h ago

I think they're just naming LLM hosting portals

2

u/Interesting-Baa 8h ago

Sorry, I was just using MS Foundry as an example of a router service, I have no idea if it was included in their test. AWS secrets were the bait for the test. They're very quiet about which services they tested. I can only find a direct mention of OpenRouter in the Dragos one, possibly because that was the only service which did the right thing when Dragos told them about the breach. Maybe they're holding off on naming others until they're sure that something or nothing has been done about it?

21

u/GooberMcNutly 21h ago

The real question is who is filtering and retaining those keys and urls? Do the gateway engines know and don't care? Is it being done in house?

And people give them access to their banks and investments for "advice"...

70

u/Practical-Custard-64 21h ago

Also in the news: "Water makes things wet."

Does anyone really think anything entered on line stays private these days?

32

u/Kyouhen 21h ago

This is why I'm increasingly moving away from cloud storage and apps associated with all of them. I've been using Obsidian more for note-taking these days and run the sync off my NAS at home. My files, my data, my choice how it's used. (And since it's all just markdown if I ever decide I don't trust Obsidian I can just take my toys and find another sandbox)

25

u/WalterCavendish 20h ago

I first read this as "Researcher plants 2 meter canaries into the AI supply chain" and was very confused as to why you would need such big birds.

1

u/PeppermintEvilButler 2h ago

I read it as 2 million canaries and was trying to figure out what in the plant they were trying to make the canaries find 

7

u/manonthemiddle 20h ago

Every middleman in that pipeline has an access log and zero incentive to delete anything.

27

u/SpecificExtension105 17h ago

Gee a product designed from the start on stolen data, steals data!!!  Why ANYONE would use Ai for anything important is baffling to me.

3

u/iamthe0ther0ne 16h ago

This isn't about the AI models. The problem surfaced with using it through third-party options like OpenRouter. 

But you know, why bother reading tge article when you can make a knee-jerk anti-AI comment to the headline for that sweet, sweet karma?

0

u/Rezolithe 7h ago

For some it's so useful it outweighs the privacy concern

Would you like to ask the modern akashic record how to do something real fast or sit there and figure it out yourself. Pair this with short form video rotting peoples brains and they go right to it

11

u/nailbunny2000 20h ago

The amount of access people are giving to 3rd parties is wild to me.

4

u/odin_the_wiggler 21h ago

Pretty glad I quit using AWS and Vercel for my projects. This also makes me think whatever GitHub is paying their security team will never be enough.

10

u/VVrayth 9h ago

If you thought anything with AI was private, you are very, very dumb.

13

u/ketosoy 20h ago • Bravo

This should be read with a skeptical eye towards yellow journalism and self promotion:  2 million tests, but no stats on what % lead to leakage; 3 providers claimed to be compromised, but not named; the “research” was published only to their own blog.

As best I can tell this is the original work, and it doesn’t have much more detail than the ars article:   https://www.dragos.com/blog/ai-supply-chain-security-canary-tokens

1

u/platinumarks 10h ago

OpenRouter, which was likely the main marketplace affected, emailed customers: AtlasCloud, Novita and StreamLake have told us they forwarded OpenRouter requests for MiniMax models to MiniMax.

11

u/Holzkohlen 18h ago

How very shocking! I thought keeping AI completely unregulated could never lead to any problems whatsoever.

8

u/MichaelFusion44 21h ago

We are all f@cked

2

u/thismorningscoffee 21h ago

Those poor birds /s

1

u/Human1649 19h ago

Are there any other sources that contains names or more data on what routes were compromised or violating ZDR?

1

u/platinumarks 10h ago

Per OpenRouter, who was one of the marketplaces affected: AtlasCloud, Novita and StreamLake have told us they forwarded OpenRouter requests for MiniMax models to MiniMax.

1

u/ExF-Altrue 13h ago

Fascinating read!

1

u/toddklindt 19h ago

I wish the Dragos article would have given names. I'm a heavy user of AI, but directly with OpenAI and Microsoft. If I read this right, the providers they tested are not first party providers like that. I inferred from the advice at the end that's the case, but I would have loved to see the names of some of the places that exposed data so that I'm 100% certain I'm not dealing with them.

2

u/illkeepthatinmind 17h ago

Yeah, like how is this actionable if not naming names, except for ceasing all 3rd party inference providers.

0

u/Tbone_Trapezius 20h ago

“Here’s my password”

“Hey somebody used my password!”