How to Recover from a Hacked Website Event

Any fellow webmaster you may ask who is beyond the novice stage will agree that one of his top priorities will always be keeping his websites secure. However, the number of exploits and tools available to hackers are so vast, and software technologies evolving so rapidly, that it is very possible, maybe likely, that you will experience a hacked website.

When addressing such an event, it can be helpful to have a short checklist of tasks to perform in your recovery process. Doing the right things in the right order will be key to maximise your chances of successful and complete recovery, as well as mitigation of future events.

Your ToDo List

Your ToDo list will contain 2 types of tasks: Preparation tasks and Action tasks. Preparation tasks make NO CHANGES to your web site or any related or underlying components AT ALL.

It is essential to clearly understand this point, because your preferred FIRST action MUST be make sure that the hacker has no way to continue accessing the system; ANY OTHER action that changes the web site may alert the hacker that he has been discovered before his access has been blocked, and you do not want to trigger the hacker into either perpetrating MORE damage, or covering his tracks.

Remember: once the event has happened, it must be treated not only as a reason to fix, but equally as a motivation to harden and secure.

• Prepare: Reaction plan
• Prepare: Battle sheet
• Action: Take your system offline
• Prepare: Clone your system to a testbed or staging server
• Prepare: Scan your website for vulnerabilities; identify and confirm suspected intrusion point
• Action: Fix the vulnerability
• Action: Bring the fixed version of the site back online; whenever possible, you should redeploy the sanitized version of your website to a clean OS/Web Server setup
• Prepare: Monitor your new and improved website
• Prepare: Make a Reaction Plan for FUTURE events.

Prepare: Reaction Plan

Keep calm and focused, and acknowledge that the event may have happened some time ago; it will not matter if you react after 30 seconds or after 15 minutes.

Once your web site has been hacked, taking a few minutes to think things through to get the most out of the incident is absolutely necessary. Take the time to write down all the steps you need to take to:

• Preserve the current situation of the web site and its underlying data to give you the best chance to eventually understand how the breach was enacted
• Avoid making rapid changes that will alert the hacker to cover his tracks or cause more damage
• Collect information and tools necessary to administer the web site and related components.

Prepare: Battle Sheet

If you are going into battle, you must make sure your armoury is populated with a complete arsenal of weapons. Basically, this means having at hand:

• Access details and relative credentials for your web site
• Filenames and paths for any configuration files which may contain information that governs access to the web site itself, or to other systems which your web site connects to or integrates with, such as:
  • database access details and credentials
  • third-party systems integration details and relative credentials or API keys or security tokens
• Access details and relative credentials for the OS or platform underlying your web site
• If you are managing the underlying hardware, access details and credentials for the “lights-out” access tools (DRAC, ILO, KVM, local or remote console, etc)
• Telephone numbers, email addresses, and support portal information for any vendor providing you with the web server and any of the underlying layers, including any related identification tokens or PINsTelephone numbers, email addresses for the developer resources you will engage for any eventual code changes or other programmatic fixes to your website.

Action: Take your System Offline

Your first action should be to take your system offline. The benefits for this are:

• Limits any additional damage the hacker may inflict on your website
• Prevents the hacker from covering his tracks, or hiding any evidence which may lead you back to him, or to his methods, or both
• Prevents your system from inflicting additional damage on other people, or accessing or damaging data or scripts on other websites (spam, Cross-site Scripting, and so on).

Before you can take your system offline, make sure that you have carefully compiled your Battle Sheet with all the necessary information for you to continue accessing the system, even though it will be offline.

Prepare: Clone your System to a TestBed or Staging Server

Some of your investigation tasks will involve running tests on your website to understand where and how it can be penetrated (hence the coined phrases “pen test” and “pen testing“). Some of these tests can be aggressive, and either cause damage to the web site, or to its underlying data, or in this case possibly to any artefacts left on the system by the hacker which could lead you to his identity or methodology or both.

To eliminate the effects of these risks, you will clone your system to a completely separate testbed or staging server. This testbed must in turn also be isolated from any public IP Address space to make sure your cloned system cannot be reached from the outside, and that your cloned system cannot itself access or affect third parties. Any systems used to test the cloned system would connect to the same private IP Address space as the clone for the purpose of running the tests.

Prepare: Scan your Website for Vulnerabilities

Once your testbed is populated with a clone of your website and related components, you can safely test your system.

You would typically use a two-pronged approach, running simultaneously:

• Scan your entire website for ALL possible points of entry
• Look for this event’s specific point of entry.

Scan your Entire Website Using Automation

The first channel of investigation would be to use a system audit approach – Using a reputable Web Vulnerability Scanner, launch a scan on your cloned testbed, which should perform the following:

• Identify misconfiguration of web server and related software
• Old and known vulnerable software versions (web server software and operating system patches typical suspects)
• Insecure security access methods (weak passwords or cypher mechanisms, for example)
• Inappropriate access to filesystem locations (such as directory visibility and permission issues)
• …and a very large battery of specific tests looking for specific known vulnerabilities.

This scanner will provide you with a list of vulnerabilities which your website is susceptible to. This type of scan can take a considerable amount of time, particularly for large and complex websites. You can use this “waiting” time to…

Look for a Point of Entry Manually

It is critical to find the actual point of entry that triggered this event. Even if you were to do a complete restore from a backup that would eliminate any damage caused by the hacker, it is only reasonable to assume that the hacker can very rapidly get back in using the exact same vehicle.

Where to look? This is not an exhaustive list, and indeed as you grow in experience, and get a deeper familiarity with the specific systems you are managing, you will be able to build a more comprehensive and specific list for your needs. The list below, however, presents a usable starting point. Do keep in mind that this particular list’s investigative actions are performed on the real web server, not on the testbed, so for the time being it is important that your manoeuvres are limited to investigation and information collection – do not make any changes for the time being.

• Get a list of running processes and look for suspicious running applications or services
  • if you have a previous clean list to compare to this would be very helpful
  • identifying the file path for a suspicious running program may help you identify the point of entry
  • identifying the user under which a suspicious program is running, or the user or group that owns the program, may help you identify the point of entry.
• If your system has been hacked, there is a VERY large possibility that the hacker may have introduced new files or changed already existing files on your system
  • if you know the date of your last change to your system, you may simply get away with checking all the files on your system to find those which have been modified or created after your last changes;
  • if your web server’s file system is static, it may be useful to compare it to a previous documented file system dump; if you do not have one, then your final review may want to consider this step for any future policies revolving around web site security
  • if not already installed, you may consider deploying software to your web server that regularly checks file system contents against a known clean file system list (typically called a baseline); this concept is called HIDS, or Host-based Intrusion Detection System; you can start by reading up about AIDE, syschangemon, Mugsy, Samhain, as a starting selection of open source tools to implement on your web server
  • check out the user or group that owns the created or modified files since this may be helpful to identify the point of entry
  • carefully check out the contents of created or modified files; the contents added or changed may help you identify the perpetrator and/or his methods, or possibly help you understand if your website is being used as a platform to launch an attack on other third party websites
• Look at the log files generated by the operating system, the web server software, and any other software also used by your web site (database servers for example).

Hopefully, your painstaking investigation of log files, running processes, and anomalous filesystem content will have allowed you to identify the entry point

Compare Manual Results with Automated Results

If you have identified one or more suspect points of entry, it can be interesting and instructional to compare your findings with the list of vulnerabilities identified by the automated web vulnerability scan.

Action: Fix the Vulnerabilities

Using the information gained from your manual investigation of files, processes, and logs, together with the (hopefully short) list of vulnerabilities compiled by the web vulnerability scanner, you can now proceed to implement the code improvements or server updates and hardening to prevent such an event from repeating itself, and plug any other weaknesses, generally making your web site safer. Summarised briefly:

• Stop any foreign processes
• Remove any extraneous files or executables
• Replace changed files and data sets with fresh or sanitised versions.

This might seem obvious, but it is critically important to check any fixes implemented by repeating the automated scan, thereby confirming that the fixes are in fact effective.

Action: Bring your Fixed Web Site Online

Now that the suspect files and process have been stopped and eliminated, and the points of entry secured, you can bring your web site back online.

If you have ANY doubt about having fully contained and reversed the work of the hacker, you should deploy a new web server host and redeploy your sanitized website and ancillary components to this new host.

Prepare: Monitor your Website

Your website is back up and running. Do not breathe too easy, however, because you need to make sure that your fixes are effective. In particular, you should devise some way (by monitoring log messages for example) to have an early-warning system of a hacker trying to get into your system via the same access vector as the event you have just fixed.

If the hacker does manage to get back in, then either you have not identified the entry point correctly, or you have not identified all of the them, or the fix is inadequate; in any case, you will need to repeat the process to again recover from this latest hack event.

Prepare: Reaction Plan for Future Events

After you have completely recovered from this event and completed all the tasks related to this event, you should revisit this section, review the Reaction Plan you made for this event, and use it to build a recovery procedure for any future event.

This will allow you to more effectively and efficiently handle your next event.