Recently I’ve been involved with a project where I needed to perform some security hardening on Amazon Web Services EC2 instances running Ubuntu Server 12.04, so I used this excellent guide as a starting point, then I added, removed and modified things as needed.
I decided to take those procedures and modify them for Ubuntu Server 14.04 now that this new LTS version has been released. Some of the procedures from 12.04 no longer need to be performed, and some needed to be changed. The following guidelines are what I’ve ended up with. You might find these guidelines useful to varying extents on other Linux distributions, but there will be potentially very significant differences depending on which distro you’re using.
Assume that all these operations need to be performed as root, which you can either do with sudo or by logging in as root first. (I’ve noticed that Ubuntu users seem particularly averse to logging in as root, apparently preferring instead to issue an endless series of commands starting with sudo, but I’m afraid that kind of extra hassle is not for me, so I just log in as root first.)
Harden SSH
I generally regard it as a very sensible idea to disable any kind of root login over SSH, so in /etc/ssh/sshd_config change PermitRootLogin to no.
If SSH on your servers is open to the world then I also advise running SSH on a non-standard port in order to avoid incoming SSH hacking attempts. To do that, in /etc/ssh/sshd_config change Port from 22 to another port of your choice, e.g. 1022. Note that you’ll need to update your firewall or EC2 security rules accordingly.
After making changes to SSH, reload the OpenSSH server:
service ssh reload
Limit “su”access to administrators only
It generally seems like a sensible idea to make sure that only users in the sudo group are able to run the su command in order to act as (or become) root:
dpkg-statoverride --update --add root sudo 4750 /bin/su
Improve IP security
Add the following lines to /etc/sysctl.d/10-network-security.conf to improve IP security:
# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Disable source packet routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
# Block SYN attacks
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5
# Log Martians
net.ipv4.conf.all.log_martians = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
# Ignore Directed pings
net.ipv4.icmp_echo_ignore_all = 1
Load the new rules:
service procps start
PHP hardening
If you’re using PHP, these are changes worth making in /etc/php5/apache2/php.ini in order to improve the security of PHP:
- Add
exec,system,shell_exec, andpassthrutodisable_functions. - Change
expose_phptoOff. - Ensure that
display_errors,track_errorsandhtml_errorsare set toOff.
Apache hardening
If you’re using Apache web server, it’s worth making sure you have the following parameters set in /etc/apache2/conf-enabled/security.conf to make sure Apache is suitably hardened:
ServerTokens Prod
ServerSignature Off
TraceEnable Off
Header unset ETag
FileETag None
For these to take effect you’ll need to enable mod_headers:
ln -s /etc/apache2/mods-available/headers.load /etc/apache2/mods-enabled/headers.load
Then restart Apache:
service apache2 restart
Install and configure ModSecurity
If you’re using Apache, the web application firewall ModSecurity is a great way to harden your web server so that it’s much less vulnerable to probes and attacks. Firstly, install the necessary packages:
apt-get install libapache2-mod-security2
Prepare to enable the recommended configuration:
mv /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf
Then edit /etc/modsecurity/modsecurity.conf:
- Set
SecRuleEnginetoOnto activate the rules. - Change
SecRequestBodyLimitandSecRequestBodyInMemoryLimitto16384000(or higher as needed) to increase the file upload size limit to 16 MB.
Next, install the Open Web Application Security Project Core Rule Set:
cd /tmp
wget https://github.com/SpiderLabs/owasp-modsecurity-crs/archive/master.zip
apt-get install zip
unzip master.zip
cp -r owasp-modsecurity-crs-master/* /etc/modsecurity/
mv /etc/modsecurity/modsecurity_crs_10_setup.conf.example /etc/modsecurity/modsecurity_crs_10_setup.conf
ls /etc/modsecurity/base_rules | xargs -I {} ln -s /etc/modsecurity/base_rules/{} /etc/modsecurity/activated_rules/{}
ls /etc/modsecurity/optional_rules | xargs -I {} ln -s /etc/modsecurity/optional_rules/{} /etc/modsecurity/activated_rules/{}
To add the rules to Apache, edit /etc/apache2/mods-available/security2.conf and add the following line near the end, just before </IfModule>:
Include "/etc/modsecurity/activated_rules/*.conf"
Restart Apache to activate the new security rules:
service apache2 restart
Install and configure mod_evasive
If you’re using Apache then it’s a good idea to install mod_evasive to help protect against denial of service attacks. Firstly install the package:
apt-get install libapache2-mod-evasive
Next, set up the log directory:
mkdir /var/log/mod_evasive
chown www-data:www-data /var/log/mod_evasive
Configure it by editing /etc/apache2/mods-available/evasive.conf:
- Uncomment all the lines except
DOSSystemCommand. - Change
DOSEmailNotifyto your email address.
Link the configuration to make it active in Apache:
ln -s /etc/apache2/mods-available/evasive.conf /etc/apache2/mods-enabled/evasive.conf
Then activate it by restarting Apache:
service apache2 restart
Install and configure rootkit checkers
It’s highly desirable to get alerted if any rootkits are found on your server, so let’s install a couple of rootkit checkers:
apt-get install rkhunter chkrootkit
Next, let’s make them do something useful:
- In /etc/chkrootkit.conf, change
RUN_DAILYto"true"so that it runs regularly, and change"-q"to""otherwise the output doesn’t make much sense. - In /etc/default/rkhunter, change
CRON_DAILY_RUNandCRON_DB_UPDATEto"true"so it runs regularly.
Finally, let’s run these checkers weekly instead of daily, because daily is too annoying:
mv /etc/cron.weekly/rkhunter /etc/cron.weekly/rkhunter_update
mv /etc/cron.daily/rkhunter /etc/cron.weekly/rkhunter_run
mv /etc/cron.daily/chkrootkit /etc/cron.weekly/
Install Logwatch
Logwatch is a great tool which provides regular reports nicely summarising what’s been going on in the server logs. Install it like this:
apt-get install logwatch
Make it run weekly instead of daily, otherwise it gets too annoying:
mv /etc/cron.daily/00logwatch /etc/cron.weekly/
Make it show output from the last week by editing /etc/cron.weekly/00logwatch and adding --range 'between -7 days and -1 days' to the end of the /usr/sbin/logwatch command.
Enable automatic security updates
N.B. Be warned that enabling automatic updates can be potentially dangerous for a production server in a live environment. Only enable this for a server in such an environment if you really know what you are doing.
Run this command:
dpkg-reconfigure -plow unattended-upgrades
Then choose Yes.
Enable process accounting
Linux process accounting keeps track of all sorts of details about which commands have been run on the server, who ran them, when, etc. It’s a very sensible thing to enable on a server where security is a priority, so let’s install it:
apt-get install acct
touch /var/log/wtmp
To show users’ connect times, run ac. To show information about commands previously run by users, run sa. To see the last commands run, run lastcomm. Those are a few commands to give you an idea of what’s possible; just read the manpages to get more details if you need to.
Edit: I recently threw together a quick Bash script to send a weekly email with a summary of user activity, login information and commands run. To get the same report yourself, create a file called /etc/cron.weekly/pacct-report containing the following (don’t forget to make this file executable) (you can grab this from GitHub if you prefer):
#!/bin/bash
echo "USERS' CONNECT TIMES"
echo ""
ac -d -p
echo ""
echo "COMMANDS BY USER"
echo ""
users=$(cat /etc/passwd | awk -F ':' '{print $1}' | sort)
for user in $users ; do
comm=$(lastcomm --user $user | awk '{print $1}' | sort | uniq -c | sort -nr)
if [ "$comm" ] ; then
echo "$user:"
echo "$comm"
fi
done
echo ""
echo "COMMANDS BY FREQUENCY OF EXECUTION"
echo ""
sa | awk '{print $1, $6}' | sort -n | head -n -1 | sort -nr
Things I haven’t covered
There are some additional issues you might want to consider which I haven’t covered here for various reasons:
- This guide assumes your Ubuntu server is on a network behind a firewall of some kind, whether that’s a hardware firewall of your own, EC2 security rules on Amazon Web Services, or whatever; and that the firewall is properly configured to only allow through the necessary traffic. However, if that’s not the case then you’ll need to install and configure a firewall on the Ubuntu server itself. The recommended software for this on Ubuntu is ufw.
- If you’re running an SSH server then you’re often told that you must install a tool such as fail2ban immediately if you don’t want your server to be hacked to death within seconds. However, I’ve maintained servers with publicly-accessible SSH servers for many years, and I’ve found that simply moving SSH to a different port solves this problem far more elegantly. I monitor logs in order to identify incoming hacking attempts, and I haven’t seen a single one in the many years I’ve been doing this. However, using this “security by obscurity” method doesn’t mean that such an attack can’t happen, and if you don’t watch your logs regularly and respond quickly to them as I do, then you would be well advised to install fail2ban or similar as a precaution, in addition to moving your SSH server to another port as described above.
- Once you’ve hardened your server, you’re advised to run some vulnerability scans and penetration tests against it in order to check that it’s actually as invincible as you’re now hoping it is. This is a topic which requires a post all of its own so I won’t be covering it in any detail here, but a good starting point if you’re not already familiar with it is the excellent Nmap security scanner.

Hi,
I use Ubuntu 14.04 x64 and I followed your tutorial to install mod_security on my server:
sudo apt-get install libapache2-mod-security2
sudo mv /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf
sudo nano /etc/modsecurity/modsecurity.conf
Make the following options so:
SecRuleEngine On
SecRequestBodyLimit 50000000
SecRequestBodyInMemoryLimit 50000000
cd /tmp
sudo wget https://github.com/SpiderLabs/owasp-modsecurity-crs/archive/master.zip
sudo apt-get install zip
sudo unzip master.zip
sudo cp -r owasp-modsecurity-crs-master/* /etc/modsecurity/
sudo mv /etc/modsecurity/modsecurity_crs_10_setup.conf.example /etc/modsecurity/modsecurity_crs_10_setup.conf
sudo ls /etc/modsecurity/base_rules | xargs -I {} sudo ln -s /etc/modsecurity/base_rules/{} /etc/modsecurity/activated_rules/{}
sudo
ls /etc/modsecurity/optional_rules | xargs -I {} sudo ln -s
/etc/modsecurity/optional_rules/{} /etc/modsecurity/activated_rules/{}
sudo nano /etc/apache2/mods-available/security2.conf
add the following line near the end, just before :
Include “/etc/modsecurity/activated_rules/*.conf”
sudo service apache2 restart
sudo rm -rfv /tmp/*
sudo apt-get install libapache2-mod-evasive
sudo mkdir /var/log/mod_evasive
sudo chown www-data:www-data /var/log/mod_evasive
sudo nano /etc/apache2/mods-available/evasive.conf
Make it so:
DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 10
DOSEmailNotify my_emailaddress@gmail.comDOSSystemCommand "su - someuser -c '/sbin/... %s ...'"
DOSLogDir "/var/log/mod_evasive"
sudo ln -s /etc/apache2/mods-available/evasive.conf /etc/apache2/mods-enabled/evasive.conf
sudo service apache2 restart
sudo apt-get install rkhunter chkrootkit
sudo nano etc/chkrootkit.conf
RUN_DAILY=”true”
RUN_DAILY_OPTS=””
sudo nano /etc/default/rkhunter
CRON_DAILY_RUN=”true”
CRON_DB_UPDATE=”true”
sudo mv /etc/cron.weekly/rkhunter /etc/cron.weekly/rkhunter_update
sudo mv /etc/cron.daily/rkhunter /etc/cron.weekly/rkhunter_run
sudo mv /etc/cron.daily/chkrootkit /etc/cron.weekly/
sudo apt-get install logwatch
sudo mv /etc/cron.daily/00logwatch /etc/cron.weekly/
sudo nano /etc/cron.weekly/00logwatch
/usr/sbin/logwatch –output mail –range ‘between -7 days and -1 days’
sudo apt-get install acct
sudo touch /var/log/wtmp
sudo a2enmod modsecurity
sudo /etc/init.d/apache2 force-reload
However when I execute: sudo a2enmod modsecurity
lupocatttivo@octane:~$ sudo a2enmod modsecurity
ERROR: Module modsecurity does not exist!
I even tried this:
lupocatttivo@octane:~$ sudo apt-get install libapache2-mod-security2
Reading package lists… Done
Building dependency tree
Reading state information… Done
libapache2-mod-security2 is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Any help will be appreciated.
Thanks in advance.
For the a2enmod command, I think it needs a hyphen, i.e. sudo a2enmod mod-security.
Beyond that I’m not sure what your problem is exactly?
great post. But has some problems.
wrong lines: (archives doesn’t exist)
etc/modsecurity/modsecurity.conf-recommended
mv /etc/modsecurity/modsecurity_crs_10_setup.conf.example
and
/etc/modsecurity/activated_rules/{}
error example
ln: Failed to create symbolic link “/etc/modsecurity/activated_rules/modsecurity_crs_55_marketing.conf ‘File already exists
etc, etc
I’m sorry you’re experiencing problems, but I can’t really tell anything from that I’m afraid. I know that plenty of people have worked through this procedure fine, so I think it’s more likely to be something odd with your setup.
thanks. i’m trying again. But is /etc/apache2/mods-available/mod-evasive.conf or /etc/apache2/mods-available/evasive.conf ???
and in the “.conf” is: DOSLogDir /var/log/mod_evasive or DOSLogDir “/var/log/apache2/mod_evasive.log” ??
thanks again
I had to hunt this down; use “$sudo a2enmod security2”
Thanks for all, good guide hardering!!!
Thanks for this post, it’s a nice follow-up to the fanclub guide.
Hi, great post, I have a problem with the phpmyadmin the mod_security block the phpmyadmin, how can solve the problem?
Thanks
I don’t use phpMyAdmin so I’m afraid I’m not familiar with it, but googling “phpmyadmin mod_security” turns up various possibilities, so I’d suggest giving those a try and seeing how you get on.
Ok thanks
If installed all your security stuff and now i can’t connect by Webbrowser
Forbidden
You don’t have permission to access /index.php
on this server.
it seems to be a problem with modsecurity if comment it out in “/etc/apache2/mods-available/security2.conf” the server works
In which case it will probably be one of the ModSecurity rules clashing with your PHP application somehow. You’d need to go through the Apache logs in /var/log/apache to look for clues. Debugging ModSecurity can be quite tricky and time-consuming…
Good! Now your server is secure. Unplug RJ45 too!
I added this line to modsecurity.conf in /etc/modsecurity
SecRuleRemoveById 970903
This prevents it from checking for code leakage on outbound traffic, since the code isn’t sensitive that rule can safely be disabled.
Before that I also had to tell the server to accept requests using an IP address.
Like Matt Brock said, check the logs!
“Before that I also had to tell the server to accept requests using an IP address.”
Please how?
It is another rule that needs to be blocked by Id number, with SecRuleRemoveById xxxxxx, if you check the logs it will tell you what the Id number is, don’t forget to use tools like grep and tail to make log checking easier
Amazing! It’s working. I also made separately custom rule conf, where I put rules with id and then symlinked it into active_rules.
Thank you very much
hello dear daveman, please can show me how you solve the problem with modsecurity? i have error accessing webserver after modsecuty installation and configuration. i know i have to add something like SecRuleRemoveById XXXXXX. but how to do that?
thx in advance for the support
brilliant post Matt.
Thanks a lot for sharing such valuable information. I have a learned a lot about security hardening through your guide.
A quick question: How to receive process accounting report via email through bash script you have shared in the end?
The report will be sent automatically via email each week. You just need to make sure that outgoing email is set up properly on your server.
Thanks Matt but where we have to mention email id for weekly report ?
This email will get sent automatically if outgoing email is set up properly on your server. Setting up email is a fairly complex task outside the scope of this article, and it’s not something I can realistically summarise in a comment.
awesome! thank you for puting all this in one spot. I would like to add a comment about fail2ban … maybe you can edit it in some where: (further details here https://www.digitalocean.com/community/tutorials/how-to-install-and-use-fail2ban-on-ubuntu-14-04 )
apt-get install fail2ban
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
edit the things you like in /etc/fail2ban/jail.local
Great post, surely looks like great solutions for the nasty hacks out there.
Gr8 post Matt
Thanks a lot …
Hi Matt. I’ve used the tips above to setup a a few webservers now, but I always seem to run into the same issue. The php mailer function seems to fail and I’ve had to setup smtp to run in its place.
The PHP hardening section of this post doesn’t seem to reference anything to do with the mailer.
Any ideas ?
Hard to say for sure without seeing some logs/debug output/etc., but I’d guess it will probably be one of the ModSecurity rules clashing with the PHP mailer. As I mentioned in another comment, you’d need to go through the Apache logs in /var/log/apache to look for clues, and debugging ModSecurity can be quite tricky and time-consuming.
great post.
With all these logs and monitors, is there an elegant solution to having say a terminal that auto-updates the logs being viewed on a remote computer?
For example, I’d like to monitor apache and other logs in real-time or with a 5-10 minute refresh time in Terminal on the Mac. I’m a newbie to sysadmin and I have heard of corncobs etc but I’m not entirely sure how I would go about this. I constantly hear stories of “ninja sys admins” sitting with multiple monitors and at least one is a constant stream of live server data, I would like a set up like that for learning and training on my VPS.
Any suggestions would be greatly appreciated ๐
Why would you not simply run tail?
Yeah tail is your best option, the above is the most elegant solution, I follow all Matt Brock’s steps and he has never let me down, plus if he did he got us this far, can’t blame him for anything.
I wish to express my gratitude for creating this website & blog.
I’m new to Linux Ubuntu Server, and this helps me a lot to take it all in.
Keep up the great work, and i wish you the best luck as a freelancer ( i don’t think you need it, but anyway ๐
installing zip doesn’t automatically add unzip. For those who run into issues, check to see if you missed the error “missing unzip”. “apt-get install unzip” will fix it.
Could you shed some light on adding a modsecurity whitelist? Other sources point to creating a whitelist.conf file and adding exceptions there which seems elegant. However I can’t get my exceptions to be honoured…
Whenever I’ve installed zip, unzip has been installed automatically. No harm in double-checking this though, for those having problems.
Adding some info about a whitelist sounds like a good suggestion. I’ll see if I can look into that at some point.
I really appreciate your blog and this post specially. I’d like to contribute adding the firewall configuration that I use (ufw):
sudo ufw default deny (this set firewall to block everything)
sudo ufw allow from YOUR_IP to any port 22
open all ports that you need (ex: sudo ufw allow 80 or sudo ufw allow 8080/tcp etc etc)
sudo ufw enable
Hope it helps
Edit: you can also use hosts.deny and hosts.allow to add an extra security layer. For example (for ssh):
set in hosts.deny –> sshd: ALL
set in hosts.allow –> sshd: YOUR_IP
After reading all the post, I realized it’s the point 1 of your uncovered things ๐
Hi there just wanted to let you know I found this guide very useful, thank you for the good read.
Hi matt,
while loading procps, I’m getting like procps stop/waiting ..
A quick google suggests this can be caused by a bug in the package. Try upgrading your Ubuntu installation.
I don’t see how the script sends an email so …
!/bin/bash
OUTFILE=
mktemp|| exit 1EMAIL=me@gmail.com
echo "USERS' CONNECT TIMES" >> $OUTFILEecho "" >> $OUTFILE
/usr/bin/ac -d -p >> $OUTFILEecho "" >> $OUTFILE
echo "COMMANDS BY USER" >> $OUTFILE
echo "" >> $OUTFILE
users=$(cat /etc/passwd | awk -F ':' '{print $1}' | sort)
for user in $users ; do
comm=$(lastcomm --user $user | awk '{print $1}' | sort | uniq -c | sort -nr)
if [ "$comm" ] ; then
echo "$user:" >> $OUTFILE
echo "$comm" >> $OUTFILE
fi
done
echo "" >> $OUTFILE
echo "COMMANDS BY FREQUENCY OF EXECUTION" >> $OUTFILE
echo "" >> $OUTFILE
sa | awk '{print $1, $6}' | sort -n | head -n -1 | sort -nr >> $OUTFILE
if [ -s “$OUTFILE” -a -n “$EMAIL” ]; then
(
echo “Subject: $(hostname -f) – Pacct Weekly Report”
echo “To: $EMAIL”
echo “”
cat $OUTFILE
) | /usr/sbin/sendmail $EMAIL
fi
rm -f $OUTFILE
exit 0
The output from the cronjob goes to root by default, so if you’ve got root email going to yourself (which you should have) then you’ll get the output automatically. But you can do it this way if you prefer.
After successfully finishing all steps, when i login into the phpmyadmin and import one databse it shows “You don’t have permission to access /phpmyadmin/import.php on this server”. Please suggest.. How to solve this problem?
Hard to say without more info. Could be that ModSecurity is preventing access. Try temporarily turning ModSecurity back off and seeing if it fixes the problem. I did some googling and found this for a similar permissions problem, so maybe worth trying something similar: http://williefang.com/2015/06/13/phpmyadmin-tbl_replace-php-permission-problem-2/ Alternatively perhaps it was a pre-existing problem, not related to the security hardening, in which case maybe the solutions suggested here might help: http://stackoverflow.com/questions/25451954/403-forbidden-when-trying-import-or-export-phpmyadmin
Thanks a lot borther