Family History Site Notes Home PageHome

TNG Traffic Analysis 2013 - 2020

These sites have been active since 2008, so Bots and attackers have had plenty of time to find them.

BingBot, Knowledge AI, and MegaIndex account for most Bot requests since 2017. The workload for their indexing is reasonable. Bing and Google provide useful indexing for researchers. Knowledge AI and MegaIndex make almost no errors, suggesting that they follow the web links rather than using links from elsewhere.

There are a number of frequent but lesser Bots Aspiegel, Petal, Seek, and many trivial Bots. From 2015 to 2017 there was a tidal wave of other Bots, but they seem to have mostly vanished. The minor Bots make few errors.

BingBot makes a large number of bad requests which have not declined since direct 400 Bad Request responses were implemented on my site in April 2019. See the Notes below and the report to Bing staff (Bing Report).

Attacks against TNG are relatively infrequent and consequently only a small part of the processing load. Because the IP address changes for each group of requests, trying to block specific IP addresses is not useful. The important part is to prevent damage from such attacks, which TNG seems to do and in any case my front-end stops everything ugly (I hope ;-).

/Family Jan-Jun Bots: Aspiegel Bing Google Know. AI Mega Petal Seek Other
All 769,399 575,426 41,549 106,068 15,506 151,645 127,396 32,710 77,293 23,217
TNG: 613,260 547,161 40,364 91,217 10,612 150,877 126,754 32,548 76,811 7,827
400 Bad   19,714 25 19,481 9 13 9 60 55 52
Not Exist   4,388 1,278 2,753 265 35 24 10 18 22
Numeric   16,930 0 16,891 0 0 0 0 0 39
Attacks:        
%20 4,575      
SQL/Code 3,930      
A=0 2,159      
passwd 802      

Notes:


/Family GETs 2019 Oct-Dec  All Bots BingBot GoogleBot MegaIndex
All 229,385 127,841 39,320 12,562 63953
TNG (Only PHP): 209,904 121,284 35,751 11,713 63,599
400 Bad Request 11,804 6,775 6,774 0 1
TNG Not Exist 1,390 1,306 1,273 7 14
Invalid Numeric 6,181 6,179 6,179 0 0
A=0 Attack 641
SQL/Code Attack 516
passwd Attacks 187

/Family GETs 2019 Jul-Oct  All Bots BingBot GoogleBot MegaIndex
All 411,017 157,753 67,915 14,984 70,099
TNG (Only PHP): 382,817 147,797 60,531 13,827 69,681
400 Bad Request 15,252  8,062 8,058 1 0
TNG Not Exist 1,663 1,526 1,465 46 18
Invalid Numeric 7,748 7,748 7,748 0 0
A=0 Attack 1,323
SQL/Code Attack 796
passwd Attacks 106

/Family GETs 2019 June  All Bots BingBot SerpStat GoogleBot
All 74,575 33,118 20,755 9,873 1,709
TNG (Only PHP): 67,505 30,127 18,480 9,695 1,439
400 Bad Request 4,422 2,503 2,501 0 2
TNG Not Exist 554 515 487 3 16
Invalid Numeric 2,439 2,439 2,439 0 0
A=0 Attack 442
SQL/Code Attack 349
passwd Attacks 4

/Family GETs 2019 May  All Bots BingBot GoogleBot MegaIndex
All 93,390 59,875 24,563 2,621 31,940
TNG (Only PHP): 86,092 56,665 22,229 2,179 31,748
400 Bad Request 5,541 2,946 2,941 4 1
TNG Not Exist 482 419 360 52 0
Invalid Numeric 2,890 2,880 2,880 0 0
A=0 Attack 821
SQL/Code Attack 1,473
passwd Attacks 12

/Family GETs 2019 April  All Bots BingBot GoogleBot MegaIndex
All 86,782 44,354 16,843 2,038 24,358
TNG (Only PHP): 86,371 41,063 14,269 1,722 24,178
400 Bad Request 3,631 1,093 976 1 0
404 Not Found 706 301 154 15 0
TNG Not Exist 188 121 112 1 3
Invalid Numeric 983 969 969 0 0
A=0 Attack 883
SQL/Code Attack 590
passwd Attacks 98

/Family 2019 Jan-Feb-Mar  All Bots BingBot GoogleBot MegaIndex
All 292,555 160,309 103,823 14,780 39,553
TNG (Only PHP): 259,817 147,921 94,183 12,920 39,239
404 Not Found 29,134 14,710 14,543 33 0
TNG Not Exist 1,701 1,609 1,538 61 3
Invalid Request 28,658 14,486 14,478 7 0
Invalid Numeric 10,015 10,0089 9,909 0 0
A=0 Attack 3,109
SQL/Code Attack 11,112
passwd Attacks 1,558
/Family GETs 2018 All All Bots BingBot GoogleBot MegaIndex
All 1,343,878 732,808 501,132 67,220 140,324
TNG (Only PHP): 1,210,827 675,134 461,960 59,380 139,329
404 Not Found 33,454 17,054 15,427 523 0
TNG Not Exist 3,474 3,300 3,032 246 12
Invalid Request 29,488 15,292 14,782 292 0
Invalid Numeric 9,090 9,021 9,021 0 0
A=0 Attack 9,727
SQL/Code Attack 3,252
passwd Attacks 3,967

/Family GETs 2017 All  All Bots BingBot GoogleBot MJ12Bot
All 2,480,769 1,981,604 389,211 26,935 784,588
TNG (Only PHP): 2,293,373 1,875,175 349,097 23,861 782,792
404 Not Found 20,916 10,889 7,794 322 4
TNG Not Exist 1,819 1,650 828 40 0
Invalid Request 13,181 4,106 1438 34 1,460
A=0 Attack 3,607
SQL/Code Attack 4,384
passwd Attacks 813
  AhrefsBot DotBot BLEXBot YandexBot SpBot MegaIndex
All 371,000 152,850 105,958 44.615 27,009 76,417
TNG (Only PHP): 339,592 152,365 96,656 37,667 15,464 75,788
404 Not Found 26 100 875 26 4 0
Invalid Request 33 99 861 67 4 0


/Family GETs 2016 All  All Bots BingBot GoogleBot MJ12Bot
All 1,744,378 1,234,881 389,393 49,834 417,261
TNG (Only PHP): 1,542,768 1,131,364 342,222 39,246 416,870
404 Not Found 21,995 17,024 15,576 246 10
TNG Not Exist 8,640 7,772 1,743 507 5,385
Invalid Request 9,692 6,647 5,650 49 643
/personID 5,311 5,311 3,875 0 10
A=0 Attack 2,102
SQL/Code Attack 957
passwd Attacks 2
AhrefsBot YandexBot SpBot XoviBot MegaIndex
All 248,996 33,006 25,656 18,427 44,275
TNG (Only PHP): 223,273 27,852 15,548 18,162 42,910
404 Not Found 5,295 40 10 2 3
TNG Not Exist 27 84 0 0 0
Invalid Request 50 27 0 0 0
/personID 246 85 0 214 0


/Family GETs 2015 All  All Bots BingBot GoogleBot MJ12Bot
All 1,240,037 748,697 265,938 99,750 196,944
TNG (Only PHP): 1,033,182 65,6778 212,664 84,069 196,943
404 Not Found 23,533 18,971 616 3,533 3,963
TNG Not Exist 2,788 1,770 1,181 553 26
/personID 10,994 10,899 6,594 0 0
A=0 Attack 563
SQL/Code Attack 374
passwd Attacks 39
  AhrefsBot XoviBot SpBot MegaIndex
All 95,920 45,222 12,216 26,157
TNG (Only PHP): 88,319 43,314 5,330 24,143
404 Not Found 32 3 87 6
/personID 1,689 1,298 0 1,300


/Family GETs 2014 All  All Bots BingBot GoogleBot BlexBot SpBot
All 798,427 351,833 43,054 98,431 185,494 14,795
TNG (Only PHP): 660,690 307,334 26,700 87,718 181,913 6,809
404 Not Found 18,889 2,012 180 279 38 127
TNG Not Exist 744 562 389 150 3 0
* Almost none of A=0 Attack, SQL/Code Attack, passwd Attack, or /personID (only 204 for Bing).

/Family GETs 2013 All All Bots BingBot GoogleBot BlexBot
All 590,416 222,321 49,069 100,389 59,884
TNG (Only PHP): * 456,055 194,173 38,964 90,152 58,916
404 Not Found 20,776 899 236 510 15
TNG Not Exist 746 651 406 247 0
* Almost none of A=0 Attack, SQL/Code Attack, passwd Attack, or /personID (but 1,313 for Bing).