Using Python scheduler to schedule asynchronous tasks

Task scheduling is common practice in modern software infrastructure that helps shift focus away from non-critical tasks by running them at a fixed time or interval. Manual tasks, such as regular data cleanups or media conversion in batches, can easily be automated via task scheduling. Task scheduling has several benefits for an organization, including increased operational efficiency and increased resource availability for customer-facing needs.

Python is a popular programming language used for task scheduling due to its simplicity and large number of use cases. In this article, we'll walk through how to use the sched module in Python to schedule tasks.

Python scheduling using sched module

Multiple Python packages and modules can help you schedule tasks. These libraries can also repeat the execution of your script over fixed intervals to ensure, for example, that your important maintenance routines are run every day.

Some other prominent use cases of task scheduling in Python include:

• Scheduling shutdowns/restarts - You can schedule your systems to automatically shut down or restart at certain times in the day to help refresh your memory and other resources.
• Running routine cleanups - You can schedule jobs to clear your database of ghost records, such as deleted users data. You can also routinely check your file storage for duplicate files and delete them to make efficient use of your resources.
• Generating reports - You can schedule jobs to set up routine report generation, such as generating a weekly head count roster for your company.
• Batch processing - You can line up a number of operations into a batch and run them at fixed time intervals for more efficient use of your resources and time.

The sched module

Some of the most popular packages and modules available in Python for task scheduling include sched, schedule, and python-crontab. Among these, sched is a built-in, general-purpose event scheduler. It's useful for one-off tasks and is thread safe.

Here's how you can use sched to schedule three print operations with some delay in between them:

The enter() function takes three mandatory arguments:

• delay - The number of seconds to wait before executing the task (measured from the time the run() function is called)
• priority - The priority of the task (the lower the number, the higher the priority)
• action - The function to be called when the task is executed

You can also pass in two additional, optional arguments:

• argument  - A sequence that carries the positional arguments for action
• kwargs  - A dictionary that carries the keyword arguments for action

Whenever the action function is executed, it's called in the following manner:

Now, let's take a look at a practical example of scheduling that prints exchange rate values against given base currencies and schedules it to be executed multiple times:

You can save the code above in a file called app.py and run it using the following command:

You will receive a similar output:

The example above relies on a public currency exchange rate API offered by ExchangeRate-API. It prints the exchange rate values against a given base currency whenever the print_exchange_rates() function is called. This method is then scheduled to be executed several times with varying delays and base currencies.

In a real-world scenario, you would save the data from the API into a CSV file for processing or you would analyze the trends in the fetched data over time and generate reports.

However, sched does not support recurring schedules out-of-the-box, so scheduling repeated tasks using this method may not be practical. You also need to keep your host system running (in this case, it would be your terminal) to execute the scheduled tasks on time. So, it's natural to look for ways to implement the same example more conveniently.

Python scheduling using Airplane

Airplane offers a very simple and elegant solution to this problem. It provides you with a powerful platform to set up Tasks that can be executed either manually or on a recurring schedule. Tasks are single business operations or multi-step functions that anyone on your team can use. Let's take a look at how you would implement this same example using Airplane.

Prerequisites

First, create an Airplane account and install the Airplane CLI. This will help you package and deploy your script to the Airplane cloud.

Once installed, run the following command to ensure that you're logged in and ready to go:

A browser window will open, asking you to log in to your Airplane account. Once done, you can create a new folder for the Airplane Task to be initialized in:

1. Initialize a new task

Initialize a new Airplane Task by running the following command:

You will be asked for some basic information about your task. Enter in the following:

Replace the contents of the exchange_rate_job.py file with the following code snippet:

This is the same script as before, except the code pertaining to the sched module is eliminated since Airplane manages the scheduling via its platform. Also, the main method returns the results instead of printing them since Airplane can display it in a human-friendly, table-based UI.

2. Deploy your task

The next step is to deploy your script as a task on the Airplane servers. Run the following command to do so:

The Airplane CLI will now begin to deploy your task, and you'll be asked if you want to create a new task with the slug exchange_rate_job. Here's what it'll look like:

3. Test the deployed task

Once the deployment succeeds, you can head over to your Airplane dashboard where you'll see the new task in your library:

Let's click on the task. Here, you'll find options to execute and schedule it:

Click Execute task to see your task in action:

You will also notice that the output generated from your Python script is intelligently converted into a tabular format for you to easily analyze. Airplane also enables you to export this data into a CSV file right from this portal:

4. Schedule recurring tasks

Once your task has been set up and tested, it's time to schedule it. Head back to the task details page and click the New schedule button:

You will be provided with an option to choose the frequency of the task execution. For complete control, you can choose the Cron schedule option to enter the routine in cron syntax:

Once you've chosen a schedule, click the Create schedule button. The schedule will be created, and the task will be executed accordingly:

You can go back to the task details page to view the executions of the task:

You can click any of the recent runs to view its logs and output:

Unlike the traditional scheduling methods in Python, you don't need to keep your host machine running to run the scheduled tasks. Setting up a task is super easy compared to maintaining a scheduler instance and manually scheduling tasks one by one.

You can also connect your Airplane Task to a plethora of external resources in a few clicks. All in all, Airplane simplifies and enriches the tedious job of maintaining scheduled tasks.

Conclusion

Task scheduling is an important aspect of application management. However, traditional methods might be holding you back in terms of convenience and efficiency. In this article, we discussed how you can use the Python sched module to schedule tasks in a local Python runtime. We also covered how the same example can be implemented quickly and conveniently using Airplane.

Airplane provides developers with an easy way to schedule tasks efficiently and effectively. In addition to task scheduling, Airplane offers Views, which allow users to quickly build internal dashboards and UIs.

Want to test it out yourself? Sign up for a free account or say hello at [email protected]. You can also check out the Airplane blog for more scheduling-related content, including 6+ cron alternatives, how to run cron in containers, how to edit your crontab, and mastering Kubernetes Jobs and Kubernetes CronJobs.

Author: Kumar Harsh

Kumar Harsh is a software developer and a technical author based in India. When he's not working on a software problem, he enjoys learning and writing content about technical topics.