Search
Archive

You are currently browsing the archives for the Failures category.

Categories
Ad

Archive for the ‘Failures’ Category

Accidental Server Down … (and so sorry for that!)

Tuesday, September 29th, 2009

My webserver was down since 13th September. It was an accident, and I did not know that until today.
No apache2 processes alive. my web server has gone away by some reasons.

So I started a deep investigation on log files.

When I checking through auth.log files with

cd /var/log;sudo find auth.log -type f -exec grep Success {} /dev/null \;

and I found a lot of

auth.log.5.gz:Aug 29 10:46:03 Shaseki su[6944]: Successful su for nobody by root
auth.log.5.gz:Aug 29 10:46:03 Shaseki su[6949]: Successful su for nobody by root
auth.log.5.gz:Aug 29 10:46:03 Shaseki su[6951]: Successful su for nobody by root

Was my server hacked by someone…? Some clever guy attacked my server and got my root password anyhow, so that he can take full control over my machine??

At the first time I saw these messages, I was so astonished and could not say a word.
But in a few minutes I came to myself and googled that message, and I was relieved.

It was not by a unknown hacker. this is by cron. Open /etc/crontab and found that

11 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
46 10    * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
15 10    * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly 
)
50 10    1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly
 )

As you can see, the time in log messages are almost same as that in crontab.

Hmmmm, So what the hell made my apache gone away?

When I tried to restart my apache like

sudo /usr/sbin/apache2ctl -k start

I got

(98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs

,so I could not start up my apache.

Obviously, some process already hold port 80. but

lsof -i :80

returns nothing. I also tried

nestat -an

, but could not find any clues of por-80-riders.

At last, thoght this is what I did not like to, I rebooted the system.

In a few minutes, everything comes back well and, as you see, my apache2 starts running healthly again.

If you know something about such a phenomenon, please leave a comment below. Thank you in advance!