In a world of microservices and cloud-native applications, the ability to define, schedule, and monitor workflows is crucial for automating processes and tasks. Argo Workflows, an open-source, container-native workflow engine for Kubernetes, allows businesses to manage and visualize cron jobs, jobs, and tasks in an efficient and scalable manner. For this article we will look at cron job management with Argo Workflows.

Why Argo Workflows for cron job management?

Businesses today have the need to manage many cron jobs. I have heard of some having up to 1500 or more! Developers and clients may need to access and trigger these jobs. Argo workflows makes managing, monitoring and executing simple. Another topic I will tackle in this article are WorkflowTemplates. WorkflowTemplates are reusable blueprints that further simplify workflow management.

In this article I will show you how to set up an argo workflow server, a cron workflow on a GCP instance and how to leverage WorkflowTemplates. At the end of the article I will discuss how to scale from a versioning standpoint. This will be in the context of using yaml files to provision resources and deploy both to GCP. For my sanity, we will use “ArgoWF” for Argo Workflows and “cronWF” for cron workflows.

Deploying the argo server

I will briefly explain setting up a cluster on a GCP instance made simple with Google Kubernetes Engine(GKE). This assumes you have an account and all tools installed locally such as gcloud.

Create a GKE cluster
gcloud container clusters create [CLUSTER_NAME] –image-type cos

Get the credentials of the cluster
gcloud container clusters get-credentials [CLUSTER_NAME]

Create the namespace argo
kubectl create ns argo

Below is an example of the install commands, ensure that you update the command to install the correct version number

kubectl apply -n argo https://github.com/argoproj/argo-workflows/releases/download/v<<ARGO_WORKFLOWS_VERSION>>/install.yaml

Stable version as of writing this article
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.4.11/install.yaml

Accessing the UI- For simplicity we will just be accessing the ui through port-forward vs creating an ingress.
kubectl -n argo port-forward svc/argo-server 2746:2746

Our first CronWF

This code snippet lays out a CronWorkflow that is designed to run on a Kubernetes cluster using Argo Workflows. Named "hello-world-cron", it's set to activate every 5 minutes, as indicated by the schedule: "*/5 * * * *" line. The magic, however, lies in the section where it references a WorkflowTemplate. Within the workflowSpec, there's an entry point labeled "whalesay-template" and a sub-section named templateRef. This sub-section is where our cron job reaches out to leverage a pre-defined WorkflowTemplate named "hello-world-template" and specifically invokes its "whalesay" component. Instead of repeating the whole recipe, it simply says, "Use that special recipe we set aside earlier."

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: hello-world-cron
  namespace: default
spec:
  # Schedule the workflow to run every 5 minutes
  schedule: "*/5 * * * *"
  # Reference the workflow template
  workflowSpec:
    entrypoint: whalesay-template
    templates:
    - name: whalesay-template
      steps:
      - - name: use-template
          # Here, we're invoking the WorkflowTemplate we previously defined
          templateRef:
            name: hello-world-template
            template: whalesay

Apply the file to our cluster kubectl apply -f cron-workflow.yaml

In the context of Argo Workflows, a "template" (not to be confused with capitalization) denotes a specific task within a Workflow. Interestingly, it can also refer to a WorkflowTemplate under the templates field. When creating a Workflow or cronWF, it's essential to define at least one template, though in practice, multiple templates are often defined. Such a template can fall under various types: container, script, dag, steps, resource, or suspend. Additionally, these templates can be invoked through an entrypoint or referenced by other 'dag' and 'step' templates.

WorflowTemplates

The WorkflowTemplate serves as a helpful component to keep work DRY ("Don't Repeat Yourself") and efficient. At its core, a WorkflowTemplate offers a blueprint for common tasks, encapsulating pre-defined processes that can be invoked across multiple workflows or cronWFs. By centralizing the logic within WorkflowTemplates, we can ensure consistency in their automated processes while significantly reducing redundancy. When managing thousands of cron jobs, the ability to reference a single, standardized template becomes invaluable.

Here is the WorkflowfTemplate our cronWF is referencing above:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: hello-world-template
  namespace: default
spec:
  # Defining input parameters for the template
  entrypoint: whalesay
  templates:
  - name: whalesay
    # This is a container-based task
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["Hello from Workflow Template!"]

And apply it: kubectl apply -f workflow-template.yaml

Check it out in the UI!

In Conclusion

Navigating the expansive realm of cloud-native applications, Argo Workflows stands out as a robust tool to bring structure and automation to repetitive tasks. In an environment where businesses thrive on streamlined processes, tools like ArgoWF empower developers to manage cron jobs with ease. With the power of Kubernetes and GCP at our disposal, we have ventured through setting up an Argo Workflow server, deploying a simple cron workflow, and harnessing the power of WorkflowTemplates for scalability and reusability. Through hands-on examples, we've seen the intuitive nature of ArgoWF and its seamless integration with GCP's infrastructure. Thank you for joining this exploration; may your workflows be ever smooth and your automations precise.