Jobs
Contents
Index
DFControl.Jobs.Job
Base.Filesystem.abspath
Base.Filesystem.joinpath
Base.append!
Base.getindex
Base.insert!
Base.pop!
Base.push!
DFControl.Client.isrunning
DFControl.Client.last_version
DFControl.Client.outputdata
DFControl.Client.readbands
DFControl.Client.readfermi
DFControl.Client.rm_version!
DFControl.Client.switch_version!
DFControl.Client.versions
DFControl.Jobs.set_flow!
DFControl.bandgap
RemoteHPC.abort
RemoteHPC.load
RemoteHPC.save
RemoteHPC.submit
Job
DFControl.Jobs.Job
— TypeJob(name::String, structure::Structure;
calculations ::Vector{Calculation} = Calculation[],
dir ::String = pwd(),
version ::Int = last_job_version(dir),
copy_temp_folders ::Bool = false,
server ::String = getdefault_server(),
environment::String ="")
A Job
embodies a set of Calculations
to be ran in directory dir
, with the Structure
as the subject.
Keywords/further attributes
calculations
: calculations to calculations that will be run sequentially.dir
: the directory where the calculations will be run.version
: the current version of the job.copy_temp_folders
: whether or not the temporary directory associated with intermediate calculation results should be copied when storing a job version. CAUTION These can be quite large.server
:Server
where to run theJob
.environment
:Environment
to be used for running theJob
.
Job(job_name::String, structure::Structure, calculations::Vector{<:Calculation}, common_flags::Pair{Symbol, <:Any}...; kwargs...)
Creates a new job. The common flags will be attempted to be set in each of the calculations
. The kwargs...
are passed to the Job
constructor.
Job(job_dir::String, job_script="job.sh"; version=nothing, kwargs...)
Loads the job in the dir
. If job_dir
is not a valid job path, the previously saved jobs will be scanned for a job with a dir
that partly includes job_dir
. If version
is specified the corresponding job version will be returned if it exists. The kwargs...
will be passed to the Job
constructor.
RemoteHPC.load
— Methodload(server::Server, j::Job)
Tries to load the Job
from server
at directory j.dir
. If no exact matching directory is found, a list of job directories that comprise j.dir
will be returned.
Interacting with calculations
Base.getindex
— Methodgetindex(job::Job, name::String)
Returns the Calculation
with the specified name
.
getindex(job::Job, i::Integer)
Returns the i'th Calculation
in the job.
Example:
julia> job["scf"]
ERROR: UndefVarError: job not defined
julia> job[2]
ERROR: UndefVarError: job not defined
Base.push!
— Methodpush!(job::Job, calculation::Calculation) = push!(job.calculations, calculation)
Base.append!
— Methodappend!(job::Job, args...) = append!(job.calculations, args...)
Base.pop!
— Methodpop!(job::Job) = pop!(job.calculations)
Base.insert!
— Methodinsert!(job::Job, i::Int, calculation::Calculation) = insert!(job.calculations, i, calculation)
Scheduling, submission and monitoring
DFControl.Jobs.set_flow!
— Functionset_flow!(job::Job, should_runs::Pair{String, Bool}...)
Sets whether or not calculations should be scheduled to run. The name
of each calculation in the job will be checked against the string in each pair of should_runs
, and the calculation.run
will be set accordingly.
Example:
set_flow!(job, "" => false, "scf" => true)
would un-schedule all calculations in the job, and schedule the "scf" and "nscf" calculations to run.
RemoteHPC.save
— Methodsave(job::Job)
Saves the job's calculations and job.sh
submission script in job.dir
. Some sanity checks will be performed on the validity of flags, execs, pseudopotentials, etc. The job will also be registered for easy retrieval at a later stage.
If a previous job is present in the job directory (indicated by a valid job script), it will be copied to the .versions
sub directory as the previous version of job
, and the version of job
will be incremented.
RemoteHPC.submit
— Functionsubmit(job::Job)
Saves and launches job
.
DFControl.Client.isrunning
— Functionisrunning(job::Job)
isrunning(s::Server, jobdir::String)
Returns whether a job is running or not. If the job was submitted using slurm
, a QUEUED
status also counts as running.
RemoteHPC.abort
— Functionabort(job::Job)
Will try to remove the job from the scheduler's queue. If the last running calculation happened to be a Calculation{QE}
, the correct abort file will be written. For other codes the process is not smooth, and restarting is not guaranteed.
Directories
Base.Filesystem.joinpath
— Methodjoinpath(job::Job, args...)
If the job is local this is joinpath(job.dir, args...)
, otherwise it will resolve the path using the Server
rootdir.
Base.Filesystem.abspath
— Methodabspath(job::Job, args...)
If the job is local this is abspath(job.dir)
, otherwise it will resolve the abspath using the Server
rootdir.
Missing docstring for cleanup(::Job)
. Check Documenter's build log for details.
Registry
All Jobs
are stored in an internal registry the first time save(job)
is called. This means that finding all previously worked on Jobs
is as straightforward as calling load(server, Job(fuzzy))
where fuzzy
is a part of the previously saved Job
dir
. This will then return a list of Jobs
with similar directories.
Versioning
As previously mentioned, a rudimentary implementation of a Job
versioning system is implemented. Upon calling save
on a Job
, if there is already a valid job
script present in job.dir
, it is assumed that this was a previous version of the job
and the script together with all other files in job.local_dir
will be copied to a subdirectory of the .versions
directory bearing the name of the respective previous job version. After this, job.version
will be incremented by 1
signalling the new version of the current Job
.
The virtue of this system is that it is possible to roll back to a previous version after possibly making breaking changes, or to pull out previous results after further experimentation was performed.
If job.copy_temp_folders=true
all possible intermediate files inside the temporary calculation directory (i.e. "job_dir/outputs") will be copied every time the job
is saved. These can be quite large and can quickly create very large job
directories. Handle with care!
DFControl.Client.versions
— Functionversions(job::Job)
versions(server::Server, jobdir::String)
Returs the valid versions of job
.
versions(job::Job)
Returs the valid versions of job
.
DFControl.Client.last_version
— Functionlast_version(job::Job)
last_version(s::Server, jobdir::String)
Returns the last version number of job
.
DFControl.Client.switch_version!
— Functionswitch_version!(job::Job[, version::Int])
Switches the version of job
to one of the previously stored ones. It will save also the current version for future reference.
DFControl.Client.rm_version!
— Functionrm_version!(job::Job, version::Int)
rm_versions!(job::Job, versions::Int...)
Removes the specified versions
from the job
if they exist.
Archiving
After a Job
is completed, or an interesting result is achieved, it makes sense to store it for future reference. This can be achieved through the archive
function. This will take the current job, and copy it to a subdirectory (specified by the second argument to archive
) of the jobs/archived
directory inside the DFControl
config directory. The third argument is a description of this job's result.
In order to not cause huge file transfers, all the temporary directories will first be removed before archiving.
Example:
archive(job, "test_archived_job", "This is a test archived job")
To query previously archived jobs one can use load(Server("localhost"), Job("archived"))
.
Missing docstring for archive
. Check Documenter's build log for details.
Output
DFControl.Client.outputdata
— Functionoutputdata(job::Job; server = job.server, calcs::Vector{String}=String[])
Finds the output files for each of the calculations of a Job
, and groups all the parsed data into a dictionary.
DFControl.Client.readfermi
— Functionreadfermi(job::Job, outdat=outputdata(job))
Tries to read the fermi level from a valid Calculation
inside job
.
DFControl.Client.readbands
— Functionreadbands(job::Job, outdat=outputdata(job))
Tries to read the bands from a bands calculation that is present in job
.
DFControl.bandgap
— Functionbandgap(job::Job, nelec=nothing)
Calculates the bandgap (possibly indirect) around the fermi level. Uses the first found bands calculation, if there is none it uses the first found nscf calculation.
bandgap(bands::AbstractVector{Band}, n_electrons::Int)
bandgap(bands::AbstractVector{Band}, fermi::Float64)
Calculates the bandgap (possibly indirect) around the fermi level.
Environments
Environments specify the skeleton of the job script, i.e. which environment variables need to be set, which scheduler flags, etc.
Missing docstring for Environment
. Check Documenter's build log for details.