If you manage a website, you might have noticed something strange in your web analytics lately. Perhaps it’s a sudden surge of traffic at odd hours, or a disproportionate number of visitors from regions where you don’t do business.

You aren’t alone. A silent but massive data harvesting operation is currently sweeping the internet.

While search engine crawlers like Googlebot have visited websites for decades to help rank them, a new wave of aggressive bots—largely originating from China—is currently vacuuming up data across the globe. Their goal isn’t to send you traffic; it is to harvest your text, images, and proprietary data to train massive AI models.

The Evidence: What’s Hiding in Your Logs

The evidence is often hiding in plain sight within your server logs and analytics dashboards.

1. The Geographic Anomaly

For many website owners, the first sign is a geographic mismatch. If you run a local service in Southeast Asia or a blog in Europe, seeing 80% to 90% of your current active users originating from China is a major red flag.

china visitor to your website

As seen in recent analytics snapshots, traffic patterns often show a handful of genuine users dwarfed by a block of connections from China. These “users” often don’t behave like humans—they don’t scroll, click, or buy. They simply load pages, copy the data, and leave.

2. Identifying the Harvesters

When you dig into the “User Agent” strings (the ID card a browser shows when visiting a site), you might see familiar names, but often you will encounter aggressive scrapers.

One common example appearing in logs recently is: Mozilla/5.0... (compatible; TikTokSpider; ttspider-feedback@tiktok.com)

ai bot is crawling your website

While TikTokSpider (owned by ByteDance) is one of the most visible, it is merely the tip of the iceberg. The race for AI dominance in China has spurred a flood of scraping activity from various sources:

  • Big Tech Giants: Companies like Baidu, Alibaba, and Tencent running massive crawlers.

  • AI Startups: Hundreds of smaller, hungry AI companies scraping aggressively to build their own datasets.

  • Research & State Institutions: There is increasing evidence of bots associated with universities and government-backed research institutes gathering vast amounts of public data for national AI infrastructure projects.

Why This Matters to You

This isn’t just about “phantom traffic.” This scraping activity has real-world consequences for your digital property:

  • Your Data is Being Taken: Your intellectual property—articles you wrote, photos you took, and data you curated—is being used to train commercial AI products without your consent or compensation.

  • Performance Degradation: These bots often ignore “polite” crawling speeds. They can hit your website thousands of times per minute, slowing it down for real customers and driving up your server bills.

  • Distorted Metrics: It becomes impossible to make data-driven decisions when half of your “audience” is actually a server farm in Beijing or Shanghai.

The Solution: Blocking AI Bots with Cloudflare

Traditional methods like blocking IP addresses are like playing whack-a-mole; these bots rotate their IPs constantly. The most effective modern defense is using a Web Application Firewall (WAF), specifically Cloudflare.

Here is how you can use Cloudflare to stop these bots:

1. Enable “Bot Fight Mode” (Free Plan)

If you are on the free tier of Cloudflare, you have immediate access to basic protection.

  • Go to your Cloudflare Dashboard.

  • Navigate to Security > Bots.

  • Toggle “Bot Fight Mode” to ON.

  • What this does: It presents a computationally expensive JavaScript challenge to visitors that look like bots. Humans pass easily; simple scraping scripts usually fail.

2. Block “AI Scrapers and Crawlers” (Pro/Biz Plans)

Cloudflare recently released a dedicated feature specifically for this issue.

  • In the dashboard, go to Security > WAF.

  • Create a new rule or look for the “AI Scrapers and Crawlers” managed rule.

  • You can set this to Block.

  • What this does: Cloudflare maintains an updated database of known AI bots (including ByteDance/TikTok, OpenAI, Apple, Amazon, and others) and blocks them automatically.

3. Create a Custom WAF Rule in your server (Advanced)

If specific Chinese bots are bypassing the general filters, you can create a “Country + User Agent” block rule:

  • Rule: If Country equals China AND User Agent contains Spider or Bot.

  • Action: Managed Challenge or Block.

  • Note: Be careful with this if you have legitimate customers in China.

4. Update Your robots.txt

This is a text file on your server that tells bots what they are allowed to do. You can explicitly disallow known AI scrapers. example:

User-agent: TikTokSpider
Disallow: /

Note: While polite bots respect this, not all scrapers do.

Summary

The internet is currently the training ground for the next generation of Artificial Intelligence, and your website is the raw material. While you cannot stop the progress of AI, you have the right to decide if and how your data is used. By recognizing these patterns and implementing tools like Cloudflare, you can shut the door on unauthorized harvesters.