How to organize Python scripts into Airplane Tasks

An increasing number of developers are using Python scripts daily. Without a proper organizational tool, too many scripts can result in a mess of different folders, CLI commands, and individual files. Airplane can help with this!

In this article, we’ll tackle how to convert a messy folder of scripts into Airplane Tasks, and how to leverage Airplane to simplify executing the scripts you need daily.

We'll cover the following:

• Prerequisites for this tutorial
• Get set up in Airplane
• Create Airplane Tasks using the CLI
• Leverage Airplane Resources in Python tasks

By the end of this post, you should be able to organize your Python scripts into Airplane Tasks quickly and efficiently.

We'll look at two straightforward scripts running on Linux.

Prerequisites

In this tutorial, you’ll need:

• A free Airplane account
• Access to a machine running Linux
• Git, Curl, and Python 3 installed (most Linux distributions have them installed by default)
• Familiarity with basic Unix commands like chmod
• SSH public/private keys set up (check out the instructions below on how to do so)

Set up SSH authentication for Python scripts

First, create an SSH RSA public/private pair key for authentication. You can do this by using create_rsa_key.sh.

Additionally, copy the public RSA key to the remote server. If you want to use a server other than localhost, you can use ssh-copy-id to distribute your public key.

Obtain the code

You can get a copy of the repository with the following code:

Both scripts have external dependencies, which are described in a requirements.txt file to make it easier for users to install using pip.

Now, install the dependencies using a virtual environment:

What do the scripts do?

The first script collects the SHA256 checksums of the files on a remote directory into your local machine. This is helpful for keeping track of unauthorized changes on sensitive files by saving their SHA256 checksums to a safe location.

Let's run it using the following:

The output of running the script will be like the following, depending on the files on the remote server:

The second script sha256_to_table.py creates a table from the report generated by the remote_sha256.py script used earlier. Run it using this code:

This is the resulting table report of the original SHA256 checksum:

This simple two-script scenario already presents a few challenges:

• The packaging of external dependencies is documented in the requirements.txt file, but installation likely requires creating a virtual environment and installing the dependencies with a tool like setuptools
• Orchestrating the scripts’ proper execution is tedious. First, you need to call the SHA256 calculator/collector, then the table generation script. You may even need to write a third bash or a shell script to call them in the proper order

Imagine doing this for all your scripts! You might be able to invest significant time automating their deployment using a solution like setuptools, but that can still be exceedingly difficult — especially for loosely coupled scripts.

Fortunately, using Airplane can help solve these issues. Let’s go over what you need to prepare in order to create Airplane Tasks using Python scripts.

Get set up in Airplane

Let's start by ensuring that the Airplane environment is properly set up. We'll walk through installing the Airplane CLI on Linux.

Once you have created your free Airplane account, you can access the Airplane command-line interface (CLI) to interact with the server. Follow these instructions to install it properly.

Consider enabling the auto-completion shell feature. This way when you start typing a command and press the Tab key, Airplane will predict and fill in the rest of the command. The following code will enable this feature:

The next step is to authenticate with Airplane using airplane auth login, which provides you with the URL needed:

Then visit the provided URL in your browser and copy the API token as shown below:

In the terminal, type airplane auth login and then paste in the API code:

Finally, confirm that you can communicate with the server using the command airplane tasks list.

In this example, you’ll see the test tasks created on the Airplane website. Your output will show something similar to the output below after typing airplane tasks list:

Now you’re ready to start using the Airplane CLI.

Create Airplane Tasks using the CLI

The building block in Airplane is an Airplane Task. A task is a function that anyone on your team can execute. Check out this documentation for detailed information on how to create a task in Python.

To create a new task for the remote_sha256.py script, enter airplane init and then answer the questions when prompted. The output is shown below:

You now have a task file called remote_sha256.task.yaml. Time to check its contents using the following:

The task definition needs a few parameters so you can execute the script:

• --server — this tells you which machine contains the files
• --remotepath — this is the directory you want to inspect
• ~/sha256-localhost-httpd-files — this is the name of the final report with the SHA256 checksums

Include the above parameters in the following code:

Airplane scripts called from Python need a special function called main to use as an entry point.

The main function obtains the dictionary that’s used to receive parameters that you will specify when you call the script as an Airplane Task:

The refactored code is similar. For this, encapsulate the write_report method inside the main method using the following code:

The next step is to replace the original remote_sha256.task.yaml contents with the arguments needed to pass to the remote_sha256 script using the following:

You can now test the task by executing it locally using the following:

You may receive the following error message:

You can fix this by installing the Airplane SDK with pip. This isn’t necessary during testing, but it’s helpful to see how the output will look when it’s run from inside Airplane.

Now, set up the virtual environment to install the Airplane SDK using the commands below:

Then, call Airplane on the developer mode, also passing the arguments on the command line instead of letting Airplane ask for it on an interactive mode:

Type the following to run your script in developer mode:

And then you will get an output like this:

Now that it is working, let’s deploy the scripts by typing airplane deploy remote_sha256.task.yaml:

And you have now successfully defined an Airplane Task.

But what if you want to connect external systems to essential tasks in Airplane? Airplane Resources are a great way to do just that.

Leverage Airplane Resources in Python tasks

Airplane Resources allow you to quickly connect to external systems, such as databases and APIs, and use them in your tasks. Examples of resources that Airplane supports include REST API, Snowflake, Slack, and more.

In Python tasks, resources can be attached through built-ins using Python SDKs or through environment variables. Note that they must be attached explicitly through task definition or the UI.

To attach resources to tasks using the UI, go to the Build your task page. Find the Resources section and select Attach resource:

After selecting the above, use the dropdown menu under Search resources to select the desired resource to attach:

Once done, hit Update task.

Let's say the resource you'd like to attach isn't in your Airplane Resources repository yet. To create a resource using the UI, navigate back to the Build your task page and select the plus icon next to the Search resources dropdown:

This will open a window where you can select which resource you'd like to connect and attach to your existing task:

Using this process, you can connect external systems to your Airplane Tasks quickly and efficiently.

Conclusion

Using Airplane to turn useful Python scripts into tasks will help improve your productivity by organizing your scripts and eliminating repetitive tasks such as setting up virtual environments. It also allows you to control how to execute code in a production environment and define relationships between scripts.

Now you’re ready to explore some more possibilities with Airplane. With Airplane, you can create SQL-based tasks, schedule API requests, and more. If you'd like to test out Airplane's extensive features, you can sign up for a free account or say hello at [email protected].