Azure DevOps agents - self-hosted - maintenance

In continuation to the previous article where we created a self-hosted agent, today we will look into its maintenance aspects

Maintenance job:

Azure DevOps provides an out-of-the-box feature called Maintenance job to clean the stale working directories and repositories

To perform maintenance on the Azure DevOps agent server , go to "Agent Pools" from "Organization Settings".

Select the Default pool where we have self-hosted the agent

In the Settings tab, enable the agent maintenance job

choose the desired settings, especially "Days to keep unused working directory"

Important Folders and Variables:

It's necessary to know about the following folders in order to debug the agent-related issues

Pipeline.Workspace - The workspace is a directory in which the agent downloads the source,runs steps, and produces outputs

Build.ArtifactStagingDirectory - The local path on the agent where any artifacts are copied before being pushed to their destination. For example: c:\agent_work\1\a

Build.SourcesDirectory - The local path on the agent where the source code files are downloaded. For example: c:\agent_work\1\s

Build.BinariesDirectory - The local path on the agent, it's an output folder for compiled binaries. For example: c:\agent_work\1\b

Clean-up task:

In Microsoft-hosted agents, a new VM is assigned for every job so we do not have to worry about any cleanup activities

however, In self-hosted agents, we may need to use the following task to clean the folders before starting a job

- job: myJob
  workspace:
    clean: outputs | resources | all # what to clean up before the job runs

outputs: Delete Build.BinariesDirectory before running a new job.

resources: Delete Build.SourcesDirectory before running a new job.

all: Delete the entire Pipeline.Workspace directory before running a new job

Common Issues:

• Permission issues:

If you see the pipeline throwing an error in the lines of "unable to create a file or directory", "the file or directory does not exist" or "ENONT"

The most likely cause is a permission issue, for example, the files may have been created as a root user and the agent user doesn't have the necessary permissions to access those files

• Disk space issues:

Another common issue is the disk space filling up on the agent server, if this happens you may need to check what are the files being created outside the scope of the maintenance job

For example, you might be using docker and the stale containers and images may need to be cleaned

If you find out that docker is the cause of the space issue, you might have to execute the following command

docker system prune - a shortcut that prunes images, containers, and networks

Summary:

We saw about the maintenance aspects of the Azure DevOps agents, we will see a comparison between self-hosted and Microsoft-hosted agents in the next week

Thanks for reading, Goodbye until next week!