abO salOm

Last Tuesday was extremely tough. I call it BLACK TUESDAY
I spent my whole day a work trying to fix the problem, and reduce the impact of the disaster.

Let me tell you the story. It was very promise morning. I started the day by reading my email, and checking my calender to see which meeting I need to attend, and which homework I need to do. Every thing went fine till afternoon 3:00pm. All of a sudden, all internet went down, and no browsing in almost all users. It was a country wide problem, and every body started calling to check what was going on. We started our investigation by checking the DNS, and we found that it is not responding. We checked the number of requests coming to the DNS, and we found around 30k to 50k requests on each one (we have 4 DNSs in production), which is not normal. We tried to identify which IP is requesting all these requests, but unfortunately, the FW is showing normal DNS requests from internal customers (each one is requesting 5 DNS requests, which is normal). So, from the symptoms, it is clear that it is an internal DDOS attack. If this is the case, it is very very hard to fix it, since you don’t want to block all your customer !

After we tried almost everything, and we fail over to Jeddah DNSs as workaround, one idea came up by one of my friend. Do we have a stable DNS version ? We checked the version and it was ISC BIND 9.5.1 P1. The surprise was it is a vulnerable version !!

Denial of service: Remote attackers can crash vulnerable systems.

We all shocked by this fact ! we spent hours trying to investigate in the wrong direction, and we never thought about the DNS version, or security issues !! It is really very important to start checking your security before anything else. Once we know this information, we downgraded the DNS to version 9.3.6, and every thing worked just fine.

I really learned a lot from that night. Thanks to my friend Abdullah, he was the hero of the black day