Family Genealogy - Site Notes Home

TNG Traffic Analysis 2013 - 2019

A simple analysis of the Family web site traffic logs to identify problems, Bot workload, and types of security attacks.

This TNG site has been active since 2008, so the Bots have had plenty of time to find it. BingBot and GoggleBot account for most Bot requests since 2017. In 2017 and earlier there was a tidal wave of other active Bots, but they seem to have pretty much vanished. BingBot has a large number of bad requests which may decline since direct 400 Bad Request responses were implemented in April 2019.

/Family GETs 2019 Jan-Feb-Mar  All Bots BingBot GoogleBot
All 291,849 120,737 103,829 15,286
404 Not Found 31,922   10.9% 16,407   13.6% 16,130   15.5% 120   00.8%
TNG Not Exist 1,692   00.6% 1,606   01.3% 1,538   01.5% 61   00.4%
Invalid Request 28,858  09.9% 14,488   12.0% 14,478   13.9% 7   00.0%
A=0 Attack 3,109   01.1%      
%20 Attacks 10,088   03.5%      
passwd Attacks 1,558   00.5%      

Notes:

  • TNG 11 does not generate any HTML error responses. This is unfortunate because Bots need the error codes to purge junk and obsolete links from their indexes.
     
  • 404 Not Found - These include TNG Not Exist and Invalid Request before 4/15/2019. I did not have a thispagedoesnotexist.html so TNG replied with a 302 Redirect and a 404 Not Found. Also my front-end rejected requests with unknown parameters or invalid parameter values with a 302 Redirect and a 404 Not Found. This was unfortunate because Bots do not connect the 404 error to the URL that caused it. After 4/15/2019, direct error responses were implemented., so we shall see if invalid Bot requests are reduced.
     
  • TNG Not Exist - Response from TNG when an ID, Tree, etc is unknown or invalid. After 4/13/2016, these are only non-existent IDs because the front-end rejects invalid parameter formats before TNG sees them (Invalid Request). After 4/15/2019, these became direct 404 Not Found.
     
  • Invalid Request - The front-end rejects requests that are invalid, have unknown parameters, or parameters with invalid values. For example, personID=123 or 123xxx  instead of I00123. This includes simple format errors, but also includes may simple and complex attacks against the TNG code usually via SQL queries. Many of the attacks are complex, and may be thousands of characters. This reduces TNG and SQL workload for attack requests. Implemented 4/13/2016  Example.txt. After 4/15/2019 these give a direct 400 Bad Request response
     
  • A=0 Attack - A brief but frequent attack, where a valid TNG call is made, usually followed by two attempts with "'A=0" after a valid Id or Tree parameter (e.g. tree=Smith'A=0). A new IP address is used for each three-request group. The User Agent is always the same and the Referrer is the same as the request. Example.txt
     
  • %20 Attacks - Requests using encoded characters such as %20 for space. Sometimes only "&" is used instead of ";"as the parameter separator. Usually in combination with code-insertion attacks against SQL. Even when they appear to be a valid request, they do not seem to come from actual customers or Bots.
     
  • passwd Attacks - Attempts to retrieve the UNIX passwd file by trying to back up from the Web file space using parameter such as "../../../../passswd#". An attack usually involves many variations of the attempt.
      
  • BingBot - BingBot is much more active than Google in 2018+ and has a much higher invalid request rate. Many of these use numerical personIDs and tree parameters for unknown reasons (I have never used numbers, especially for tree). Up to 4/15/2016, neither TNG nor the front-end used direct error responses, so perhaps BingBot never eliminated such invalid requests. After 4/15/2019, this site was changed to use a direct 40x responses, so we will see if this error rate decreases.
     
  • All Bots - Counted by searching for "bot/" whereas individual Bots counted by full name, so All Bots may be less than the sum of listed individual bots when individual Bots do not have "bot/" in their User_Agent string.
     
  • Admin Attacks - Strangely there are almost no attempts to access TNG administrative tools.
/Family GETs 2018 All All Bots BingBot GoogleBot
All 1,340,810 592,037 501,146 72,395
404 Not Found 40,338   3.0% 21,022   3.5% 18,745   3.7% 1,187   1.6%
TNG Not Exist 3,467   0.3% 3,288   0.6% 3,032   0.6% 246   0.3%
Invalid Request 29,488   2.2% 15,292   2.6% 14,782   2.9% 292   0.4%
A=0 Attack 9,727   0.7%      
%20 Attacks 5,742   0.4%      
passwd Attacks 3,967   0.3%      

/Family GETs 2017 All  All Bots BingBot GoogleBot MJ12Bot
All 2,477,468 1,904,975 389,223 33,878 784,588
404 Not Found 29,719 17,803 9,645 736 2,384
TNG Not Exist 1,818 1,644 828 40 641
Invalid Request 13,181 4,106 1442 34 1,460
A=0 Attack 3,603        
%20 Attacks 3,962        
passwd Attacks 813        
  AhrefsBot DotBot BLEXBot YandexBot SpBot
All 371,032 152,850 105,963 44.627 27,042
404 Not Found 720 141 3,453 310 192
TNG Not Exist 23 5 5 100 0
Invalid Request 33 99 864 25 70

/Family GETs 2016 All  All Bots BingBot GoogleBot MJ12Bot
All 1,738,519 1,189,975 389,407 58,922 417,212
404 Not Found 39,189 31,032 15,576 1,499 6,856
TNG Not Exist 8,629 7,772 1,743 507 5,386
Invalid Request 9,196 6,601 5,650 24 592
A=0 Attack 1,935        
%20 Attacks 902        
passwd Attacks 4        
AhrefsBot YandexBot SpBot XoviBot
All 249,010 33,018 25,674 18,425
404 Not Found 5,295 1,383 140 328
TNG Not Exist 27 84 0 0
Invalid Request 50 27 47 0
 
/Family GETs 2015 All  All Bots BingBot GoogleBot MJ12Bot
All 1,200,186 700,411 264,225 90,237 196,946
404 Not Found 23,533 18,971 7,660 4,167 3,963
TNG Not Exist 2,788 1,770 1,181 553 26
A=0 Attack 563
passwd Attacks 39        
  AhrefsBot OviBot SpBot
All 95,934 44,767 12,180
404 Not Found 1,785 1,307 87
TNG Not Exist 0 15 0


/Family GETs 2014 All  All Bots BingBot GoogleBot BlexBot
All 1,444,8747 349,187 43,068 109,863 185,494
404 Not Found 18,889 2,012 1,060 1,625 87
TNG Not Exist 744 562 389 155 5
 
/Family GETs 2013 All  All Bots BingBot GoogleBot BlexBot
All 583,972 220,704 49,084 136,964 59,884
404 Not Found 20,776 3,658 37 3,355 37
TNG Not Exist 746 651 406 260 0



Page modified: 19 Apr 2019 14:28:31 -0700

[Back to Top Send comments or questions to Contact Me. Home