TNG Traffic Analysis 2013 - 2020

These sites have been active since 2008, so Bots and attackers have had plenty of time to find them.

BingBot, Knowledge AI, and MegaIndex account for most Bot requests since 2017. The workload for their indexing is reasonable. Bing and Google provide useful indexing for researchers. Knowledge AI and MegaIndex make almost no errors, suggesting that they follow the web links rather than using links from elsewhere.

There are a number of frequent but lesser Bots Aspiegel, Petal, Seek, and many trivial Bots. From 2015 to 2017 there was a tidal wave of other Bots, but they seem to have mostly vanished. The minor Bots make few errors.

BingBot used to make a large number of bad requests. This stopped several years ago. See the Notes below and the report to Bing staff (Bing Report).

Attacks against TNG are relatively infrequent and consequently only a small part of the processing load. Because the IP address changes for each group of requests, trying to block specific IP addresses is not useful. The important part is to prevent damage from such attacks, which TNG seems to do and in any case my front-end stops everything ugly (I hope ;-).

/Family	Jan-Jun	Bots:	Aspiegel	Bing	Google	Know. AI	Mega	Petal	Seek	Other
All	769,399	575,426	41,549	106,068	15,506	151,645	127,396	32,710	77,293	23,217
TNG:	613,260	547,161	40,364	91,217	10,612	150,877	126,754	32,548	76,811	7,827
400 Bad		19,714	25	19,481	9	13	9	60	55	52
Not Exist		4,388	1,278	2,753	265	35	24	10	18	22
Numeric		16,930	0	16,891	0	0	0	0	0	39
Attacks:
%20	4,575
SQL/Code	3,930
A=0	2,159
passwd	802

Notes:

• These TNG sites return a simple "thanks for indexing us" response to TNG requests from Bots for pages that are set for no index and no follow, and do not actually call TNG. This greatly reduces server workload for pages that add no information to indexes.
   
• TNG 11 as released does not generate HTML error responses for invalid parameters or unknown data. This is unfortunate because Bots may need the error codes to purge junk and obsolete links from their indexes. Beginning April 2019 local code changes give direct 40x responses as described below.
   
• TNG Not Exist (404 Not Found after 4/4/2019; 410 Gone after 5/2/2019)
  When a person, etc. is unknown, including invalid parameter formats, TNG, as released, responds with a 302 Redirect to thispagedoesnotexist.html. The redirected request fails with 404 Not Found. After 4/4/2019, this site was changed so that any unknown person or data gets a direct 404 Not found response. Then after 5/2/2019 changed to reply 410 Gone because most were for deleted items.
   
• 404 Not Found (After 4/15/2019 only for TNG Not Exist; After 5/2/2019 not used for TNG)
  This  includes TNG Not Exist and Invalid Requests before 4/16/2019. After 4/13/2016, the front-end rejected requests with unknown parameters, invalid values, or disallowed requests with a 302 Redirect to "/URI=" resulting in a 404 Not Found.
   
• Invalid Request (400 Bad Request after 4/15/2019)
  The local front-end rejects requests that are mal-formed, have unknown parameters, have invalid values, or use disallowed requests. This includes simple format errors, but also includes many simple and complex attacks against the TNG code, usually via SQL injection queries. Many of the attacks are complex, and may be thousands of characters. The front-end reduces TNG and SQL workload for attack requests, reduces compromise risk, and was first implemented 4/13/2016.
   
• Invalid Numeric (400 Bad Request after 4/15/2019)
  Simple numeric values in the personID or tree parameters have never been used here. Requests using these are strange because a number is used in one or the other, but not both! Also strange, because it only occurs with BingBot in 2017 and later, and not before.
     
• %20 Attack
  Most attacks involve special characters in the tree and ID parameters. This category includes attacks using %20 (space) and other special characters, presumably hoping they will cause errors in TNG. SQL and passwd attacks are not included in this count, they are counted separately.
   
• A=0 Attack
  A brief but frequent attack, where a valid TNG call is made, usually followed by two attempts with "'A=0" in a personID or tree parameter (e.g. tree=Smith'A=0). A new IP address is used for each three-request group. The User Agent is always the same and the Referrer is the same as the request.
   
• SQL/Code Attack
  Requests using inserted SQL such as SELECT, UNION, and AND. Also other code attached to parameters. This includes both simple and long, complex URL attacks.
   
• passwd Attacks
  Attempts to retrieve the UNIX passwd file by trying to back up from the Web file space using parameter such as "../../../../passswd#". An attack usually involves many variations of the attempt.
    
• All Bots
  Counted by searching for "bot" in the HTTP User Agent string plus "Crawler", "MegaIndex", and "Knowledge AI".
   
  
• BingBot
  BingBot is much more active than Google in 2018+ and has a much higher invalid request rate. Many of these use numerical personID or tree parameters, which have never been numeric (Invalid Numeric). After 4/16/2019, this site was changed to use a direct 400 Bad Request  response. See he report to Bing staff (Bing Report).
   
• Admin Attacks - Strangely there are almost no attempts to access TNG administrative tools.
   
• 403 Denied (After 4/16/2019)
  Some exotic problems, probably internal to TNG, respond with 403 Denied.
   
• personID - Request without TNG command beginning with /personID=...

/Family GETs 2019	Oct-Dec	All Bots	BingBot	GoogleBot	MegaIndex
All	229,385	127,841	39,320	12,562	63953
TNG (Only PHP):	209,904	121,284	35,751	11,713	63,599
400 Bad Request	11,804	6,775	6,774	0	1
TNG Not Exist	1,390	1,306	1,273	7	14
Invalid Numeric	6,181	6,179	6,179	0	0
A=0 Attack	641
SQL/Code Attack	516
passwd Attacks	187

/Family GETs 2019	Jul-Oct	All Bots	BingBot	GoogleBot	MegaIndex
All	411,017	157,753	67,915	14,984	70,099
TNG (Only PHP):	382,817	147,797	60,531	13,827	69,681
400 Bad Request	15,252	8,062	8,058	1	0
TNG Not Exist	1,663	1,526	1,465	46	18
Invalid Numeric	7,748	7,748	7,748	0	0
A=0 Attack	1,323
SQL/Code Attack	796
passwd Attacks	106

/Family GETs 2019	June	All Bots	BingBot	SerpStat	GoogleBot
All	74,575	33,118	20,755	9,873	1,709
TNG (Only PHP):	67,505	30,127	18,480	9,695	1,439
400 Bad Request	4,422	2,503	2,501	0	2
TNG Not Exist	554	515	487	3	16
Invalid Numeric	2,439	2,439	2,439	0	0
A=0 Attack	442
SQL/Code Attack	349
passwd Attacks	4

/Family GETs 2019	May	All Bots	BingBot	GoogleBot	MegaIndex
All	93,390	59,875	24,563	2,621	31,940
TNG (Only PHP):	86,092	56,665	22,229	2,179	31,748
400 Bad Request	5,541	2,946	2,941	4	1
TNG Not Exist	482	419	360	52	0
Invalid Numeric	2,890	2,880	2,880	0	0
A=0 Attack	821
SQL/Code Attack	1,473
passwd Attacks	12

/Family GETs 2019	April	All Bots	BingBot	GoogleBot	MegaIndex
All	86,782	44,354	16,843	2,038	24,358
TNG (Only PHP):	86,371	41,063	14,269	1,722	24,178
400 Bad Request	3,631	1,093	976	1	0
404 Not Found	706	301	154	15	0
TNG Not Exist	188	121	112	1	3
Invalid Numeric	983	969	969	0	0
A=0 Attack	883
SQL/Code Attack	590
passwd Attacks	98

/Family 2019	Jan-Feb-Mar	All Bots	BingBot	GoogleBot	MegaIndex
All	292,555	160,309	103,823	14,780	39,553
TNG (Only PHP):	259,817	147,921	94,183	12,920	39,239
404 Not Found	29,134	14,710	14,543	33	0
TNG Not Exist	1,701	1,609	1,538	61	3
Invalid Request	28,658	14,486	14,478	7	0
Invalid Numeric	10,015	10,0089	9,909	0	0
A=0 Attack	3,109
SQL/Code Attack	11,112
passwd Attacks	1,558

/Family GETs 2018	All	All Bots	BingBot	GoogleBot	MegaIndex
All	1,343,878	732,808	501,132	67,220	140,324
TNG (Only PHP):	1,210,827	675,134	461,960	59,380	139,329
404 Not Found	33,454	17,054	15,427	523	0
TNG Not Exist	3,474	3,300	3,032	246	12
Invalid Request	29,488	15,292	14,782	292	0
Invalid Numeric	9,090	9,021	9,021	0	0
A=0 Attack	9,727
SQL/Code Attack	3,252
passwd Attacks	3,967

/Family GETs 2017	All	All Bots	BingBot	GoogleBot	MJ12Bot
All	2,480,769	1,981,604	389,211	26,935	784,588
TNG (Only PHP):	2,293,373	1,875,175	349,097	23,861	782,792
404 Not Found	20,916	10,889	7,794	322	4
TNG Not Exist	1,819	1,650	828	40	0
Invalid Request	13,181	4,106	1438	34	1,460
A=0 Attack	3,607
SQL/Code Attack	4,384
passwd Attacks	813

	AhrefsBot	DotBot	BLEXBot	YandexBot	SpBot	MegaIndex
All	371,000	152,850	105,958	44.615	27,009	76,417
TNG (Only PHP):	339,592	152,365	96,656	37,667	15,464	75,788
404 Not Found	26	100	875	26	4	0
Invalid Request	33	99	861	67	4	0

/Family GETs 2016	All	All Bots	BingBot	GoogleBot	MJ12Bot
All	1,744,378	1,234,881	389,393	49,834	417,261
TNG (Only PHP):	1,542,768	1,131,364	342,222	39,246	416,870
404 Not Found	21,995	17,024	15,576	246	10
TNG Not Exist	8,640	7,772	1,743	507	5,385
Invalid Request	9,692	6,647	5,650	49	643
/personID	5,311	5,311	3,875	0	10
A=0 Attack	2,102
SQL/Code Attack	957
passwd Attacks	2

	AhrefsBot	YandexBot	SpBot	XoviBot	MegaIndex
All	248,996	33,006	25,656	18,427	44,275
TNG (Only PHP):	223,273	27,852	15,548	18,162	42,910
404 Not Found	5,295	40	10	2	3
TNG Not Exist	27	84	0	0	0
Invalid Request	50	27	0	0	0
/personID	246	85	0	214	0

/Family GETs 2015	All	All Bots	BingBot	GoogleBot	MJ12Bot
All	1,240,037	748,697	265,938	99,750	196,944
TNG (Only PHP):	1,033,182	65,6778	212,664	84,069	196,943
404 Not Found	23,533	18,971	616	3,533	3,963
TNG Not Exist	2,788	1,770	1,181	553	26
/personID	10,994	10,899	6,594	0	0
A=0 Attack	563
SQL/Code Attack	374
passwd Attacks	39

	AhrefsBot	XoviBot	SpBot	MegaIndex
All	95,920	45,222	12,216	26,157
TNG (Only PHP):	88,319	43,314	5,330	24,143
404 Not Found	32	3	87	6
/personID	1,689	1,298	0	1,300

/Family GETs 2014	All	All Bots	BingBot	GoogleBot	BlexBot	SpBot
All	798,427	351,833	43,054	98,431	185,494	14,795
TNG (Only PHP):	660,690	307,334	26,700	87,718	181,913	6,809
404 Not Found	18,889	2,012	180	279	38	127
TNG Not Exist	744	562	389	150	3	0

* Almost none of A=0 Attack, SQL/Code Attack, passwd Attack, or /personID (only 204 for Bing).

/Family GETs 2013	All	All Bots	BingBot	GoogleBot	BlexBot
All	590,416	222,321	49,069	100,389	59,884
TNG (Only PHP): *	456,055	194,173	38,964	90,152	58,916
404 Not Found	20,776	899	236	510	15
TNG Not Exist	746	651	406	247	0

* Almost none of A=0 Attack, SQL/Code Attack, passwd Attack, or /personID (but 1,313 for Bing).