6

So today a client of mine sent out a marketing newsletter to around 140k clients that included a link to our web app.

What happened next was my web app experienced a flood of traffic (over 9000 requests in 15 minutes) from Microsoft-owned IP addresses in the range 40.94././ requesting that specific page on my site. This took the app down for all my clients until I managed to restart it.

It seems like the scan took place regardless of whether a user clicked on the link or not, as there are no other IP addresses in the request logs for the same url during this period.

So my question is, was this Microsoft pre-emptively scanning that link as it was delivered to newsletter subscribers? Or does the scan only happen when the link is clicked - I've found conflicting information on this, and as mentioned I see no other IP address requests during this period.

And secondly, how can I stop this from happening in the future - is my only option to blacklist this IP range, or are there other strategies?

Gary
  • 742
  • 8
  • 20
  • You can't stop this because how useful would be a virus scanner that have an option that disables it (would make it very easy for real attackers to bypass the scanner)? The easiest solution would be to use only links in mass mailings that lead to static pages or pages that require very little effort for the server (regarding CPU and network bandwidth). Alternative host the server in the cloud and before sending a mass mailing "buy" the appropriate number of cloud systems to host your web site. – Robert Apr 20 '21 at 13:23
  • 1
    @Robert. This is hosted in Azure, but it would be ridiculous (and impossible) to have to up (and pay for) my cloud-infrastructure every time one of my clients links to my site in a newsletter. This seems like blatant abuse by an automated system. – Gary Apr 20 '21 at 13:40

2 Answers2

4

So for anyone struggling with something similar I can confirm that Microsoft pre-emptively scans the links inside a received email before it lands in the recipients inbox.

The effect of this is that if a huge newsletter is sent to hundreds of thousands of recipients, Microsoft effectively triggers a wave of traffic to your server.

It would appear the only solution is to black-list their range of IPs, or ensure you have some throttling mechanism in place.

Gary
  • 742
  • 8
  • 20
  • IP blacklisting is a possibility, but you should also try to see if there is a programmatic way of doing it, such as with a robots.txt file or by blocking a specific useragent. This can make it an easy implementation for all platforms, if anyone wants to block them from their server in the future. – Xiddoc Apr 29 '21 at 10:53
  • @Xiddoc. Unfortunately blocking via user-agent does not work as they mimic a normal browser user-agent and do not specify that it's Safe Links related. I've slightly adjusted our built-in software throttle to look for this kind of pattern of traffic and throttle accordingly. But at present, I don't have a better way around it. – Gary Apr 30 '21 at 11:08
  • https://stackoverflow.com/a/65095191/3952494 - this seems interesting, have you verified it? – Klapsa2503 Nov 16 '21 at 09:47
  • @Klapsa2503 Interesting, but it seems blocking HEAD requests may not be optimal in some cases either. I will have a look at blocking them at some point and provide feedback. – Gary Nov 16 '21 at 15:53
  • 1
    So referencing https://security.stackexchange.com/questions/62811/should-i-disable-http-head-requests it would seem disabling HEAD requests is not a solution. – Gary Nov 16 '21 at 16:00
1

One of the solution as mentioned in the other answer is to block the range of ip addresses that belong to Microsoft Safelink in order to prevent the scans from accessing the website.

Other solution might be to use JS Challenge such as this available in Cloudflare. With such a solution each user has to go through a website that first verifies if he/she is using a real browser and only if that is the case he/she is redirected to the target website.

Such a JS Challenge can be enabled only for those accessing website from links in the email so that anyone using browser to directly access a website won't be affected

Klapsa2503
  • 829
  • 10
  • 33
  • This is an interesting idea, but how would one control that this only occurs for links in an email? Given that some clients may just drop a link to your app into a newsletter for instance? – Gary Nov 17 '21 at 10:05
  • 1
    It's all about reducing unwanted traffic, not about eliminating it completely. Links in an email can always have some unique parameter such as `?source=email` and you can enable Cloudflare rules based on properties present in the url. Of course you need to remember to add those parameter to each link in the email but in most cases we develop newsletters from our systems so you could add some newsletter post processor that whould modify the links. Some marketing campaign applications automatically add such a custom parameters for analytics purpose such as (utm_*) params – Klapsa2503 Nov 18 '21 at 10:52