How to check website or url health periodically and automatically by using python script?

Dec 30, 2024

1. Purpose

In this post, I will demo how to create a python script to check a url(website/webservice)‘s health periodically and automatically .

2. Environment

Python 2.x or 3.x
Linux system

3. The solution

3.1 The environment

Suppose we hava a web service which provides some sort of services to users, now we want it to run always and if it’s not healthy, we should restart the service automatically.

The service URL prefix is:

http://localhost:8080/myservice

And we also have a restart script for this service located at:

/opt/myservice/restart.sh

3.2 Add a health check endpoint for our service

For health check purpose, we should add a dedicated endpoint to be accessed by the health check script, it should be fast , here is our implementation in java language:

  @RequestMapping(method = RequestMethod.GET,value = "hello",headers = "Accept=application/json")
  public @ResponseBody String hello() {
    return "world";
  }

It means that if we caccess the below url:

http://localhost:8080/myservice/hello

We should get this result(http response status code equals 200):

world

Otherwise , the status code should not be 200 , just as the picture shows:

It health check fails, the restart script should be run immediately.

3.3 Setup the logger and imports for python script

Before coding the business, we should setup the environment for our script, just as follows:

import time
import os
import logging
from logging.handlers import TimedRotatingFileHandler

#setup the loggers for daily logging file rolling
logger = logging.getLogger()
logger.setLevel(logging.INFO)
file_handler = TimedRotatingFileHandler("healthcheck_myservice.log", when="midnight", interval=1)
file_handler.suffix = "%Y%m%d"
ch = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)
ch.setFormatter(formatter)
logger.addHandler(file_handler)

# global variables.
THE_URL = "http://localhost:8080/myservice/hello"
RESTART_SH = '/opt/myservice/restart.sh'
SLEEP_INTERVAL = 60

In the above code, we use the logging module in python to setup a daily rolling file handler for our program, it would create a log file everyday with the suffix year-month-day. You can view more information about the TimedRotatingFileHandler from this website.

The parameter when="midnight" in TimedRotatingFileHandler means that it would Roll over at midnight.

3.4 Implement the health check function with python

First, we need a function to check the status of the service, just as follows:

def health_check(url):
    try:
        logger.info("start access url %s" % (url))

        import urllib
        a=urllib.urlopen(url) #this is python2 version, the python3 compatible version is listed below

        if a:
            if a.getcode()==200:
                lines = a.readlines()
                if lines is not None and len(lines)==1:
                    if lines[0]=='world':
                        logger.info("health ok")
                        return True
            else:
                logger.info("warning code:"+str(a.getcode()))
    except Exception as e:
        logger.error('health check failed %s ' % (str(e)))
    logger.info("health not ok")
    return False

You can see that , the above function would return True only when the service response code is 200 and the responsecontent is world, or else, it would return False to the caller.

For python 2.x , we can use:

import urllib
urllib.urlopen(url)

For python 3.x , you can replace the urllib.urlopen with follows:

import urllib.request
with urllib.request.urlopen(url) as response:
   html = response.read()

3.5 Implement the periodically checking logic

Now we should create a function to run the health check periodically and restart the service automatically if the check fails.

def check_periodically(url,failed_action_func):
    while True:
        try:
            time.sleep(SLEEP_INTERVAL)
            logger.info("check awake, start check")
            if not health_check(url):
                failed_action_func()
        except Exception as e:
            logger.error('check periodically failed %s ' % (str(e)))
        pass

def restart_server():
    do_script(RESTART_SH)
    pass

def do_script(the_scripts):
    logger.info("=========start script %s======"%the_scripts)
    result = os.popen(the_scripts).read()
    logger.info(result)
    logger.info("=========end   script %s======" % the_scripts)
    pass

3.6 The main function

At last, we should provide the main function to execute the whole script:

if __name__ == '__main__':
    check_periodically(THE_URL,restart_server)
    pass

3.7 Save the script and execute it in background

Then we save the above script as healthcheck_myservice.py, and execute in background as follows:

$ nohup python healthcheck_myservice.py > healthcheck_myservice.out &

Then we can check the log as follows:

$ tail -f healthcheck_myservice.log

We can get these logs:

2021-06-16 15:54:23,775 - root - INFO - health ok
2021-06-16 15:55:23,834 - root - INFO - check awake, start check
2021-06-16 15:55:23,834 - root - INFO - start access url http://localhost:8080/myservice/hello
2021-06-16 15:55:23,839 - root - INFO - health ok
2021-06-16 15:56:23,899 - root - INFO - check awake, start check
2021-06-16 15:56:23,899 - root - INFO - start access url http://localhost:8080/myservice/hello
2021-06-16 15:56:23,906 - root - INFO - health ok

It works!

4. The whole script

#check if my service is running fine, or else, it would restart it

import urllib
import time
import os
import logging
from logging.handlers import TimedRotatingFileHandler

logger = logging.getLogger()
logger.setLevel(logging.INFO)
file_handler = TimedRotatingFileHandler("healthcheck_myservice.log", when="midnight", interval=1)
file_handler.suffix = "%Y%m%d"
ch = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)
ch.setFormatter(formatter)
logger.addHandler(file_handler)
#logger.addHandler(ch)

THE_URL = "http://localhost:8080/myservice/hello"
RESTART_SH = '/opt/myservice/restart.sh'
SLEEP_INTERVAL = 60


#perform the health check, return True if the url is accessible.
def health_check(url):
    try:
        logger.info("start access url %s" % (url))
        a=urllib.urlopen(url)
        if a:
            if a.getcode()==200:
                lines = a.readlines()
                if lines is not None and len(lines)==1:
                    if lines[0]=='world':
                        logger.info("health ok")
                        return True
            else:
                logger.info("warning code:"+str(a.getcode()))
    except Exception as e:
        logger.error('health check failed %s ' % (str(e)))
    logger.info("health not ok")
    return False

#check the health for specified period
def check_periodically(url,failed_action_func):
    while True:
        try:
            time.sleep(SLEEP_INTERVAL)
            logger.info("check awake, start check")
            if not health_check(url):
                failed_action_func()
        except Exception as e:
            logger.error('check periodically failed %s ' % (str(e)))
        pass

def restart_server():
    do_script(RESTART_SH)
    pass

def do_script(the_scripts):
    logger.info("=========start script %s======"%the_scripts)
    result = os.popen(the_scripts).read()
    logger.info(result)
    logger.info("=========end   script %s======" % the_scripts)
    pass

if __name__ == '__main__':
    check_periodically(THE_URL,restart_server)
    pass

5. Summary

In this post, I demonstrated how to use python to do health check job in linux.