Back to Posts List

Internet users in China hit with a massive DNS issue

Share this article





Posted on January 21st, 2014 by Websitepulse in WebSitePulse News, Tech

DNSToday (Jan 21, 2014) between 07:00 and 09:00 UTC mainland China was hit by a wave of network issues. A large number of both Chinese and international domains were reporting connection timeouts and were completely unaccessible. The cause for this was that for some reason a large number of domains were resolving to a single IP address - 65.49.2.178. The problem was reported as resolved in most places around 08:50. Due to the DNS caching infrastructure however a lot of users will still be affected until the cached bogus records expire. Users were suggested to use alternative DNS servers like Google's 8.8.8.8. We strongly recommend if you are using a DNS caching server in China to flush it as soon as possible to avoid DNS issues

The news sources in China first blamed it on a DNS poisoning of the root and the gTLD DNS servers. This however is not the case since the networks outside mainland China were not affected at all. Our investigation showed DNS responses from authoritative name servers coming with bogus data when queried from our servers in Beijing, Shangai and Guangzhou.

For example our first notice of the issue was when we requested the A record of static.bbci.co.uk. from ns1.thdow.bbc.co.uk (212.58.240.163), which is one of the authoritative DNS serves for bbc.co.uk, the response was

static.bbci.co.uk.      37621     IN      A 65.49.2.178

instead of  the correct record which is

static.bbci.co.uk.      900     IN      CNAME   static-bbci.bbc.net.uk.

This suggests that the response packet was modified in transit and the most likely culprit is the Golden Shield Project (also known as The Great Firewall of China). A possible reason for the misshap is that instead of blocking the 65.49.2.178 IP address all DNS queries were redirected to this IP. Ironically they actually succeeded blocking the IP by creating a massive DDOS attack from all Chinese users who were making connections to this address while trying to access different sites.

The offending IP 65.49.2.178 has an interesting story itself but I'll just put a few pointers
 - It is owned by Sophidea, Inc. registered on the address 2710 Thomes Ave Suite 884, Cheyenne, WY, 82001, US. Reuters ran a special story about this address a few years ago (www.reuters.com/article/2011/06/28/us-usa-shell-companies-idUSTRE75R20Z20110628)
 - While there is scant information regarding Sophidea, Inc I can see that they are providing hosting services to a large number of companies and the IPs from this range were frequently reported for spam
 - Hosted on the same network range is ultrasurf.us which was created to provide means to bypass the China Firewall and provide anonymous Internet access. Ultrareach Corp - the company which owns UltraSurf is registered on the same postal address in Wyoming above. There was some discussion of the actual value of the services provided by UltraSurf  (blog.torproject.org/blog/ultrasurf-definitive-review) but still their service might have something to do with the Chinese goverment wanting to restrict access to this netblock

 Update Jan 22, 2014

The widespread news that a DNS poisoning attack has taken place are largely false. While there was secondary DNS cache poisoning due to this problem the initial error came from the Great Firewall of China and its regional sub-systems. First were affected the domains with low TTL which were frequently visited by Chinese internet users like Weibo and Baidu, but not all sites were affected - for example sina.com was getting correct IPs during the whole period.

An interesting fact is that after the IP for a DNS servers was set to the bogus address 65.49.2.178, which at the moment was already unreachable we were still getting bogus DNS responses from it. Here is an example from our DNS cache logs converted for better explanation

1. Query domain.com address
2. We have cached NS records ns1.domain.com and ns2.domain.com
3. Find address for ns1.domain.com

response from a.gtld-servers.net. (92.5.6.30) - this should not happen. The gtld servers should return NS records, not A
ns1.domain.com 19613 IN  A  65.49.2.178

4.  Find address for ns2.domain.com
response from j.gtld-servers.net. (192.48.79.30) - this should not happen. The gtld servers should return NS records, not A
ns2.domain.com 34332 IN  A  65.49.2.178

5. Find address for domain.com from the above DNS server addresses (both are 65.49.2.178)
response from 65.49.2.178 - this IP was not reachable at the time. It is obviously coming from a different location pretending to be it
domain.com 36993 IN  A  65.49.2.178

 Another fact about the IP  - it was routed via Dynamic Internet Technology Inc.(dit-inc.us) which lists among its clients theepochtimes.com (banned in China) and is developing Freegate (an anti-censorship software). Their network took most of the traffic which at the time seemed like a massive DDOS attack. The estimates from some Chinese providers state that more than 200 million users were affected. This raises the question about the security of the Great Firewall. By setting similar DNS filtering rules a malicious attacker can direct the traffic of all China users to any single IP in the world effectively bringing it down in minutes. Currently China has more than 600 million internet users and for comparison the largest botnet in the world to date was estimated at around 30 million bots.

We will update this post with additional information as soon some official statement regrading the issue is made. Please let us know if you have any thoughts on this.

Glossary

DNS - Domain Name System. The global DNS infrastructure serves as an Internet address book. It allows you to find the IP address needed to connect to a domain name. For example www.websitepulse.com currently resolves to the IP address 204.232.239.198. Additionally it serves information about the email servers behind an email address (MX records), the email servers that are allowed to send email from a domain (SPF records), IPv6 addresses (AAAA records) and more. See more on DNS

DNS spoofing - an attack serving bogus DNS responses to trick users to connect to a different IP address or service. Similar to DNS cache poisoning.

DNS hijacking - an attack that compromises a trusted DNS server or the packets coming from it to redirect the users to different addresses.Also called DNS redirection.

NS records - a typical DNS record containing the domain names and/or IP addresses of the DNS servers responsible for a domain

A records - a DNS record containg the IP address(es) corresponding to a domain name

Golden Shield (The Great Firewall of China) - a large censorship and surveilance project run by the goverment of China. In addition to the capability of blocking specific domains and addresess it is supposed to be able to do  DNS filtering and redirecton and Man-in-the-middle attacks.

Websitepulse

WebSitePulse is a leading provider of advanced, independent and remote monitoring services that enable clients to increase the efficiency of their mission-critical e-business operations, and to reduce their risk of failed Internet transactions and loss of revenue.

comments powered by Disqus