abigail phoebe

8% of images on Discord are scams: automatically detecting mr. beast scams on Discord

Have you ever seen those crypto-scams on Discord where they post 4 images, usually starting with Mr. Beasts Twitter profile?

They look something like this (though this is a parody, obviously).

I kinda went somewhat viral on Twitter recently over my solution to these scams, so by popular demand, here's a write up on how it works!

I got annoyed of having to delete these in my communities (I run a Discord partnered server, as well as around 10 others), so I looked around for the available solutions people have already made. There were many, though none quite felt good enough for me.

One of the most popular available solutions to these scams are the use of "honeypot" channels. These are really simple: a channel is created by the bot, and if any user sends a message in them, they are banned. This works well when bots just spam in every single channel they have access to, though it also catches innocent users who don't read the channel's information, not to mention how easy it is for scam developers to bypass, as all they would need to do is develop a simple check if a channel is a honeypot channel and then avoid it.

Another popular tool I came across relied on perceptual hashing to detect the scams, this similarly had some shortcomings. There are several types of perceptual hashing algorithms, though all we tried had the same issues; at its core a perceptual hash comparison will only match if the images are either exact matches, or very very similar matches.

Since the Mr. Beast scam images use photos of monitors with different brightness, cropping, angle of the photo being taken, etc, to a computer they look entirely different. While we as humans can easily see that two different images of a monitor are the same, a computer cannot if it is using a simple algorithm such as perceptual hashing. Instead it can only match to images in the same scam-batch, which once flagged by a human, is already too late and has been sent hundreds of times across Discord.

This is not to mention other issues with common perceptual hashing algorithms such as a fairly small bit-space, which leads to two completely different images being matched as the same image (something we ran into a few times during testing).

As both of these popular detection methods were lackluster at best, and since I am a computer scientist, I figured I'd try and make my own (why spend 5 seconds doing a task which I can spend 5 days automating?).

Introducing Scrump Anti-scam: A roughly 10-stage detection pipeline that not only has a near 100% match rate, but is yet to have any credible false positives, can adapt in real-time to new scam images, and can also run entirely locally on a fairly small Hetzner server while handling over 200 messages per second.

Before I get into the details of this post, I want to quickly highlight that none of your images are ever saved by Scrump, nor are they sent to third-parties. Scrump follows EU privacy laws, and is hosted locally in Europe.

I will not be going into detail on the exact algorithms and detection pipeline Scrump uses, as doing so would help scammers bypass its detection. However I will outline at a high level what we are doing so that anyone with a bit of background knowledge on the subject can reasonably figure it out.

As I already mentioned, perceptual hashing is just not good enough for our task, so we needed something different. We initially thought about running OCR (optical character recognition, i.e. read text from an image) on every image and then checking the text, which works perfectly!..... for small servers. When you only need to scan a few images per minute, OCR is great! However, when building at Scrump's scale, without using a paid, privacy-issue raising, and unknown external API to perform the OCR, it ends up eating the entire CPU of our server, since uh... well, we get "a few" images per second.

What if we used perceptual hashing to detect similar enough images, and then did OCR? This was another idea we tried, though this would still miss some known images and also sent too many irrelevant images to OCR (thus blowing up the CPU...).

Eventually, we settled upon a custom implementation of an algorithm developed by Meta to detect CSAM as our first-pass filter (to clarify: this is all running fully on-device, absolutely no data is ever shared with Meta or any other third parties). This filter simply tells us "this is definitely not a scam" or "i don't know".

From there, any image flagged "i don't know" is sent into a further pipeline to check against known images using a variety of hashing algorithms: frequency-space matching, vector-space differentials, and some secret checks. This stage filters the "i don't know" into "absolutely 100% a scam", "maybe a scam", and "safe".

From here, an image flagged as safe is ignored, any 100% scam is blocked, and any "maybe a scam" is flagged for yet another round of checks.

This second-to-last set of checks are our super-duper-secret (note: actually really simple but security by obscurity) checks that filter the image into "not a scam", "still not too sure", and "definitely a scam" buckets.

As to be expected, "not a scam" is discarded, and "definitely a scam" is blocked. The "still not too sure" part is where the real power of Scrump comes into play. Due to our scale, when these scammers start sending out new scam images, we see them across multiple accounts in multiple servers in relatively quick succession. If the same image is sent by different people in different servers within a short time span, and they both reach this final stage, the image is automatically promoted to a known scam image & punishments are retroactively applied (to clarify: only servers that have opted into using anti-scam are used in this step).

If you read that and went some form of "Wait, that sounds scary, what if it's a false positive?" - that is a fair question; alongside all of our previous checks in the pipeline for an image to ever get to this stage, we take into account *a lot* more than what I have gone into in this post, and we have not had any false positives reach the third stage, let alone the fourth, and certainly not across multiple servers. In the unlikely event a false positive does reach this stage, reach out and let us know! We can mark it and immediately stop further detection's, and revert previous punishments.

I wish I was able to give a more technical breakdown of our algorithm & scam-detection pipeline (as holy hell we put a lot of effort into it), but with the popularity of my recent Twitter post on this topic, I feel as though the people making these scams will be playing close attention to anything I say regarding how it works in an attempt to circumvent it. This is the same reason I will not be open sourcing the algorithm.

Scrump Anti-scam has now been running for a couple days across some of the largest servers on Discord, and I can proudly say that we have caught all of our targeted scams, automatically found over 200 different variants, and have had no reports of false positives.

An interesting statistic we found over the past few days since deploying this feature is that of the servers that have enabled our scam detection, an average of 8% of all images sent were scams... which is insane!

So... yeah! Hope you enjoyed reading this! add Scrump to your servers today :) Until next time!

Blog reads