Family Genealogy - Site Notes Home

TNG Traffic Analysis 2013 - 2019

These TNG sites have been active since 2008, so the Bots and attackers have had plenty of time to find it.

BingBot, GoggleBot, and MegaIndex account for most Bot requests since 2017. The workload for their indexing is reasonable. Bing and Google provide useful indexing for researchers. MegaIndex causes almost no errors. This suggests that they are following the site and not URLs from other sources. From 2015 to 2017 there was a tidal wave of other Bots, but they seem to have mostly vanished.

BingBot has a large number of bad requests which have not declined since direct 400 Bad Request responses were implemented in April 2019. See the Notes below and the report to Bing staff (Bing Report).

/Family 2019 Jan-Feb-Mar  All Bots BingBot GoogleBot MegaIndex
All 292,555 160,309 103,823 14,780 39,553
TNG: 259,817 147,921 94,183 12,920 39,239
404 Not Found 29,134 14,710 14,543 33 0
TNG Not Exist 1,701 1,609 1,538 61 3
Invalid Request 28,658 14,486 14,478 7 0
Invalid Numeric 10,015 10,0089 9,909 0 0
A=0 Attack 3,109        
SQL/Code Attack 11,112        
passwd Attacks 1,558        
/Family GETs 2019 April  All Bots BingBot GoogleBot MegaIndex
All 86,782 44,354 16,843 2,038 24,358
TNG: 86,371 41,063 14,269 1,722 24,178
400 Bad Request 3,631 1,093 976 1 0
404 Not Found 706 301 154 15 0
TNG Not Exist 188 121 112 1 3
Invalid Numeric 983 969 969 0 0
A=0 Attack 883        
SQL/Code Attack 590        
passwd Attacks 98        
/Family GETs 2019 May  All Bots BingBot GoogleBot MegaIndex
All 93,390 59,875 24,563 2,621 31,940
TNG: 86,092 56,665 22,229 2,179 31,748
400 Bad Request 5,541 2,946 2,941 4 1
TNG Not Exist 482 419 360 52 0
Invalid Numeric 2,890 2,880 2,880 0 0
A=0 Attack 821        
SQL/Code Attack 1,473
passwd Attacks 12        
/Family GETs 2019 June  All Bots BingBot SerpStat GoogleBot
All 74,575 33,118 20,755 9,873 1,709
TNG: 67,505 30,127 18,480 9,695 1,439
400 Bad Request 4,422 2,503 2,501 0 2
TNG Not Exist 554 515 487 3 16
Invalid Numeric 2,439 2,439 2,439 0 0
A=0 Attack 442        
SQL/Code Attack 349  
passwd Attacks 4        
/Family GETs 2019 July  All Bots BingBot GoogleBot MegaIndex
All 103,842 57,120 22,623 2,021 31,967
TNG: 94,689 54,136 20,329 1,692 31,788
400 Bad Request 4,219 2,661 2,658 0 0
TNG Not Exist 476 439 417 16 6
Invalid Numeric 2,619 2,618 2,618 0 0
A=0 Attack 237        
SQL/Code Attack 235
passwd Attacks 82        

Notes:

  • TNG 11 as released does not generate HTML error responses for invalid parameters or unknown data. This is unfortunate because Bots need the error codes to purge junk and obsolete links from their indexes. In April 2019 local code changes give direct 40x responses as described below.
     
  • TNG Not Exist (404 Not Found after 4/4/2019; 410 Gone after 5/2/2019)
    When a person, etc. is unknown, including invalid parameter formats, TNG as released responds with a 302 Redirect to thispagedoesnotexist.html. The redirected request fails with 404 Not Found. After 4/4/2019, TNG was changed so that any unknown person or data gets a direct 404 Not found response. Then after 5/2/2019 changed to 410 Gone because all such were deleted items.
     
  • 404 Not Found (After 4/15/2019 only for TNG Not Exist; After 5/2/2019 not used for TNG)
    This  includes TNG Not Exist and Invalid Request before 4/16/2019. After 4/13/2016, the front-end rejected requests with unknown parameters, invalid values, or disallowed requests with a 302 Redirect to "/URI=" resulting in a 404 Not Found.
     
  • Invalid Request (400 Bad Request after 4/15/2019)
    The local front-end rejects requests that are mal-formed, have unknown parameters, have invalid values, or use disallowed requests. This includes simple format errors, but also includes many simple and complex attacks against the TNG code, usually via SQL queries. Many of the attacks are complex, and may be thousands of characters. The front-end reduces TNG and SQL workload for attack requests, reduces compromise risk, and was first implemented 4/13/2016.  Example.txt.
     
  • Invalid Numeric (400 Bad Request after 4/15/2019)
    Simple numeric values in the person or tree parameters have never been used and are invalid. Requests using these are strange because a number is used in one or the other, but not both! Also strange, because it only occurs with BingBot in 2017 and later, and not before.  Example.txt
     
  • personID - Request without TNG command beginning with /personID=...
     
  • A=0 Attack
    A brief but frequent attack, where a valid TNG call is made, usually followed by two attempts with "'A=0" in a valid ID or Tree parameter (e.g. tree=Smith'A=0). A new IP address is used for each three-request group. The User Agent is always the same and the Referrer is the same as the request. Example.txt
     
  • SQL/Code Attack
    Requests using inserted SQL such as SELECT, UNION, and AND. Also other code attached to parameters. This includes both simple and long, complex URL attacks.
     
  • passwd Attacks
    Attempts to retrieve the UNIX passwd file by trying to back up from the Web file space using parameter such as "../../../../passswd#". An attack usually involves many variations of the attempt.
      
  • BingBot
    BingBot is much more active than Google in 2018+ and has a much higher invalid request rate. Many of these use numerical person or tree parameters, which have never been numeric (Invalid Numeric). Up to 4/15/2016, neither TNG nor the front-end used direct error responses, so perhaps BingBot never eliminated such invalid requests. After 4/16/2019, this site was changed to use a direct 400 Bad Request  response, so we will see if this error rate decreases. See he report to Bing staff (Bing Report).
     
  • All Bots
    Counted by searching for "bot/" whereas individual Bots counted by full name, so All Bots may be less than the sum of listed individual bots when individual Bots do not have "bot/" in their User_Agent string. MegaIndex is included in All bots despite no "bot" in its name.
     
  • MegaIndex - Crawler MegaIndex.ru. Very few errors, so apparently following web links accurately. Apparently active since 2015.
     
  • Admin Attacks - Strangely there are almost no attempts to access TNG administrative tools.
     
  • 403 Denied (After 4/16/2019)
    Some exotic problems, probably internal to TNG, respond with 403 Denied.
/Family GETs 2018 All All Bots BingBot GoogleBot MegaIndex
All 1,343,878 732,808 501,132 67,220 140,324
TNG: 1,210,827 675,134 461,960 59,380 139,329
404 Not Found 33,454 17,054 15,427 523 0
TNG Not Exist 3,474 3,300 3,032 246 12
Invalid Request 29,488 15,292 14,782 292 0
Invalid Numeric 9,090 9,021 9,021 0 0
A=0 Attack 9,727        
SQL/Code Attack 3,252        
passwd Attacks 3,967        

/Family GETs 2017 All  All Bots BingBot GoogleBot MJ12Bot
All 2,480,769 1,981,604 389,211 26,935 784,588
TNG: * 2,293,373 1,875,175 349,097 23,861 782,792
404 Not Found 20,916 10,889 7,794 322 4
TNG Not Exist 1,819 1,650 828 40 0
Invalid Request 13,181 4,106 1438 34 1,460
A=0 Attack 3,607        
SQL/Code Attack 4,384        
passwd Attacks 813        
  AhrefsBot DotBot BLEXBot YandexBot SpBot MegaIndex
All 371,000 152,850 105,958 44.615 27,009 76,417
TNG: ** 339,592 152,365 96,656 37,667 15,464 75,788
404 Not Found 26 100 875 26 4 0
Invalid Request 33 99 861 67 4 0
* 860 Invalid Numeric for Bingbot but no other Bot.  ** Only a few TNG Not Exist for these Bots.

/Family GETs 2016 All  All Bots BingBot GoogleBot MJ12Bot
All 1,744,378 1,234,881 389,393 49,834 417,261
TNG: *** 1,542,768 1,131,364 342,222 39,246 416,870
404 Not Found 21,995 17,024 15,576 246 10
TNG Not Exist 8,640 7,772 1,743 507 5,385
Invalid Request 9,692 6,647 5,650 49 643
/personID 5,311 5,311 3,875 0 10
A=0 Attack 2,102        
SQL/Code Attack 957        
passwd Attacks 2        
AhrefsBot YandexBot SpBot XoviBot MegaIndex
All 248,996 33,006 25,656 18,427 44,275
TNG: *** 223,273 27,852 15,548 18,162 42,910
404 Not Found 5,295 40 10 2 3
TNG Not Exist 27 84 0 0 0
Invalid Request 50 27 0 0 0
/personID 246 85 0 214 0
*** No Invalid Numeric

/Family GETs 2015 All  All Bots BingBot GoogleBot MJ12Bot
All 1,240,037 748,697 265,938 99,750 196,944
TNG: 1,033,182 65,6778 212,664 84,069 196,943
404 Not Found 23,533 18,971 616 3,533 3,963
TNG Not Exist 2,788 1,770 1,181 553 26
/personID 10,994 10,899 6,594 0 0
A=0 Attack 563
SQL/Code Attack 374        
passwd Attacks 39         
  AhrefsBot XoviBot SpBot MegaIndex
All 95,920 45,222 12,216 26,157
TNG: * 88,319 43,314 5,330 24,143
404 Not Found 32 3 87 6
/personID 1,689 1,298 0 1,300
* No TNG Not Exist for these Bots.

/Family GETs 2014 All  All Bots BingBot GoogleBot BlexBot SpBot
All 798,427 351,833 43,054 98,431 185,494 14,795
TNG: 660,690 307,334 26,700 87,718 181,913 6,809
404 Not Found 18,889 2,012 180 279 38 127
TNG Not Exist 744 562 389 150 3 0
* Almost none of A=0 Attack, SQL/Code Attack, passwd Attack, or /personID (only 204 for Bing).

/Family GETs 2013 All  All Bots BingBot GoogleBot BlexBot
All 590,416 222,321 49,069 100,389 59,884
TNG: * 456,055 194,173 38,964 90,152 58,916
404 Not Found 20,776 899 236 510 15
TNG Not Exist 746 651 406 247 0
* Almost none of A=0 Attack, SQL/Code Attack, passwd Attack, or /personID (but 1,313 for Bing).


Page modified: 02 Aug 2019 12:00:54 -0700

[Back to Top Send comments or questions to Contact Me. Home