Prevent Unnecessary Task Execution

By fingerprinting locally generated files and their sources

We can avoid unnecessary workload of tasks to reduce the time taken. Meaning that the tasks will only run when there are changes in the required files.

For example, when you work on a Node.JS application, you may have a task to install dependencies and build the application.

• npm install - run this command when there are changes in the package.json file.
• npm run build - run this command where there are changes in any **/*.js files.

version: '3'
 
tasks:
  # if the package-lock.json is deleted, the task will not run again
  npm:install:without-generates:
    cmds:
      - npm install
    sources: # it will compare the checksum to see whether the files got any changes
      - package.json
 
  # if the package-lock.json is deleted, the task will run again
  npm:install:
    cmds:
      - npm install
    sources: # it will compare the checksum to see whether the files got any changes
      - package.json
    generates: # this will be the files generated by the command, if this file is deleted, the task will run again
      - package-lock.json
  
  npm:build:
    cmds:
      - npm run build
    sources:
      - '**/*.js'
      - exclude: ignore.js
 
  python_build:
    cmds:
      - python3 -m build
    sources:
      - setup.py
      - '**/*.py'
      - exclude: tests/** # exclude all files in the tests directory
    generates:
      - dist/*.whl # generated wheel file
    method: timestamp

• sources and generates can be files or glob patterns
  • sources - it will compare the checksum to see whether the files got any changes
  • generates - this will be the files generated by the command, if this file is deleted, the task will run again
• exclude - exclude files from fingerprinting, but you have to make sure the sources are in glob format and must come after the positive glob it is negating.
• method: timestamp - If you prefer to check the timestamp of the files instead of their checksum, you can use method: timestamp. This is useful for large files where checksum calculation might be expensive.
  • method: none - Skips the validation, meaning the task will always run regardless of changes in the source files.
  • method: checksum - This is the default method, which checks the checksum of the files to determine if they have changed.
• status - You can also use the status command to check the status of the task. It will show whether the task is up to date or not. Refer Using programmatic checks to indicate a task is up to date

ubuntu@touted-mite:~/nodejsfun$ task npm:install
task: [npm:install] npm install
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: 'ansi-escapes@6.2.1',
npm WARN EBADENGINE   required: { node: '>=14.16' },
npm WARN EBADENGINE   current: { node: 'v12.22.9', npm: '8.5.1' }
npm WARN EBADENGINE }
npm WARN EBADENGINE Unsupported engine {
npm WARN EBADENGINE   package: 'marked-terminal@5.2.0',
npm WARN EBADENGINE   required: { node: '>=14.13.1 || >=16.0.0' },
npm WARN EBADENGINE   current: { node: 'v12.22.9', npm: '8.5.1' }
npm WARN EBADENGINE }
 
up to date, audited 62 packages in 723ms
 
3 packages are looking for funding
  run `npm fund` for details
 
4 moderate severity vulnerabilities
 
Some issues need review, and may require choosing
a different dependency.
 
Run `npm audit` for details.
 
# second time - no changes in package.json
ubuntu@touted-mite:~/nodejsfun$ task npm:install
task: Task "npm:install" is up to date

When you rerun the task, it will not run the npm install command as there are no changes in the package.json file. You will see a message indicating that the task is up to date.

Using programmatic checks to indicate a task is up to date

You can also use programmatic checks to indicate whether a task is up to date. In this case, you can use the status command to check the status of the task before running the commands. If the task is up to date, it will skip running the commands.

This method is quite useful when you want to create artifacts or files that are not directly related to the source files, or when you want to ensure that certain files exist before running a task.

Here is how it works: - Task always checks the status first.

1. On the first run, the status checks will fail (because the directory and files don’t exist), so the commands will execute to create them.
2. On subsequent runs, the status checks will pass (because the directory and files now exist), so the commands will not run again. The task is skipped.

version: '3'
 
tasks:
  generate-files:
    cmds:
      - mkdir directory
      - touch directory/file1.txt
      - touch directory/file2.txt
    # test existence of files
    status:
      - test -d directory
      - test -f directory/file1.txt
      - test -f directory/file2.txt

ubuntu@touted-mite:~$ task generate-files 
task: [generate-files] mkdir directory
task: [generate-files] touch directory/file1.txt
task: [generate-files] touch directory/file2.txt
 
ubuntu@touted-mite:~$ task generate-files 
task: Task "generate-files" is up to date

Combination of sources and status checks:

With the combination of sources and status checks, if either the sources change or programmatic checks fail, then the task will run again.

version: '3'
 
tasks:
  npm:install:
    cmds:
      - npm install
    sources:
      - package.json
    status:
      - test -f package-lock.json

Using programmatic checks to cancel the execution of a task and its dependencies (Preconditions Checks)

You can setup preconditions checks to run before the task commands. If any of the checks fail, the task will not run. It is very similar to the status command just that it support sh expansion. If any preconditions fail, the task and its dependencies are canceled and do not run.

version: '3'
tasks:
  generate-files:
    cmds:
      - mkdir directory
    # test existence of files
    preconditions:
      - test -f .env
      - sh: '[ 1 = 0 ]'
        msg: "This will not run because the condition is false"

Here is an example of the dependencies are canceled when the preconditions fail:

version: '3'
 
tasks:
  task-will-fail:
    preconditions:
      - sh: 'exit 1'
 
  task-will-also-fail:
    deps:
      - task-will-fail
 
  task-will-still-fail:
    cmds:
      - task: task-will-fail
      - echo "I will not run"

Difference between status and preconditions

Limiting when tasks run

Sometimes you may want to limit when tasks run based on certain conditions especially there are multiple cmds or deps in the task. In this case, you can use the run to change the behavior of the task execution.

Supported run options:

• always (default) - The task will always run regardless of the number of times it has been run.
• once - The task will only run once, even if it has been run multiple times.
• when_changed - The task will only run once for each unique set of variables passed into the task.

version: '3'
 
tasks:
  default:
    cmds:
      - task: generate-file
        vars: { CONTENT: '1' }
      - task: generate-file
        vars: { CONTENT: '2' }
      - task: generate-file
        vars: { CONTENT: '2' }
 
  generate-file:
    run: when_changed
    deps:
      - install-deps
    cmds:
      - echo {{.CONTENT}}
 
  install-deps:
    run: once
    cmds:
      - sleep 5 # long operation like installing packages

ubuntu@touted-mite:~$ task default
task: [install-deps] sleep 5
task: [generate-file] echo 1
1
task: [generate-file] echo 2
2

As you can see, the install-deps task only runs once, even though it is called multiple times in the default task. The generate-file task runs only once for each unique set of variables passed into it.