Automated application layer attacks by bots have long been used by criminals, particularly with the expansion of internet-connected devices such as surveillance cameras with low security.
However, a new report from security vendor Imperva warns some bot operators are expanding their sights by rebranding their operations as business intelligence companies selling scraped data. These operators are hiring professional data extraction experts and investing in new techniques to evade detection.
“Bad bots are trying to improve their image and appear legitimate,” Imperva says in its seventh annual report. It says so-called “bad” bots do everything from web scraping without permission, competitive data mining, personal and financial data harvesting, brute-force login, digital ad fraud, spam, transaction fraud, and more.
The biggest problem with bots is they are behind credential stuffing and brute force attacks. However, the report adds that not all bad bots are run by criminals. Some legitimate businesses scrape publicly-available data from competitors’ web sites for pricing or job-filling purposes. Others are in a murky world that may cross into price-fixing by, for example, buying event tickets from a competitor’s web site, buying up stocks of limited edition running shoes or trying to influence an election through spreading misinformation.
With such a wide definition, it’s no surprise that Imperva says the U.S. remains the “bad bot superpower” with nearly 46 per cent of bad bot traffic coming from the country. The Netherlands is the third-largest source with eight per cent, followed by Canada with 6.3 per cent.
“Good” bots, Imperva says, ensure that prospective customers can find online businesses and their products. Examples include search engine crawlers such as GoogleBot and Bingbot that, through their indexing, help people match their queries with the most relevant sets of websites
By Imperva’s definition, bad bots accounted for 24 per cent of all web traffic in 2019, up six per cent from 2018. Of that number 73.7 per cent are what Imperva calls advanced persistent bots that cycle through random IP addresses, enter through anonymous proxies, change their identities and mimic human behaviour.
Good bots made up 13 per cent of all web traffic last year.
The top five industries with the most bad bot traffic include financial services (47.7 per cent), education (45.7 per cent), IT and services (45.1 per cent), marketplaces (39.8 per cent), and government (37.5 per cent).
To combat bad bots Imperva says CIOs/CISOs should:
- Because bad bots disguise their identity by reporting their user agent as a web browser to avoid detection, block, or CAPTCHA outdated user agents and browsers, like Internet Explorer, Chrome, Firefox and Safari
- Block traffic from known bot hosting and proxy services such as Digital Ocean, Gignet, OVH Hosting and Choopa LLC. The report says Amazon is the biggest source of bad bots. However, the proportion dropped to 11.6 percent in 2019 from 18 per cent the previous year
- Protect exposed APIs and mobile apps—not just your website—and share blocking information between systems wherever possible. Protecting your website does little good if backdoor paths remain open, says the report;
- Monitor traffic sources carefully for suspicious signs such as high bounce rates and lower conversion rates from certain traffic sources, traffic spikes and failed login attempts;
- Retailers should watch for an increase in failures, or even traffic, to gift card validation pages. These can be a signal that
bots such as GiftGhostBot are attempting to steal gift card balances.
The full report is available here. Registration is required.