AWSBatch
AWSBatch.jl provides a small set of methods for working with AWS Batch jobs from julia.
Installation
AWSBatch assumes that you already have an AWS account configured with:
- An ECR repository and a docker image pushed to it [1].
- An IAM role to apply to the batch jobs.
- A compute environment and job queue for submitting jobs to [2].
Please review the "Getting Started with AWS Batch" guide and example CloudFormation template for more details.
Basic Usage
julia> using AWSBatch
julia> job = run_batch(
name="Demo",
definition="AWSBatchJobDefinition",
queue="AWSBatchJobQueue",
image = "000000000000.dkr.ecr.us-east-1.amazonaws.com/demo:latest",
role = "arn:aws:iam::000000000000:role/AWSBatchJobRole",
vcpus = 1,
memory = 1024,
cmd = `julia -e 'println("Hello World!")'`,
)
AWSBatch.BatchJob("00000000-0000-0000-0000-000000000000")
julia> wait(job, [AWSBatch.SUCCEEDED])
true
julia> results = log_events(job)
1-element Array{AWSBatch.LogEvent,1}:
AWSBatch.LogEvent("00000000000000000000000000000000000000000000000000000000", 2018-04-23T19:41:18.765, 2018-04-23T19:41:18.677, "Hello World!")
AWSBatch also supports Memento logging for more detailed usage information.
API
AWSBatch.run_batch
— Methodrun_batch(;
name::AbstractString="",
queue::AbstractString="",
region::AbstractString="",
definition::Union{AbstractString, JobDefinition, Nothing}=nothing,
image::AbstractString="",
vcpus::Integer=1,
memory::Integer=-1,
role::AbstractString="",
cmd::Cmd=``,
num_jobs::Integer=1,
parameters::Dict{String, String}=Dict{String, String}(),
) -> BatchJob
Handles submitting a BatchJob based on various potential defaults. For example, default job fields can be inferred from an existing job definition or an existing job (if currently running in a batch job).
Order of priority from highest to lowest:
- Explict arguments passed in via
kwargs
. - Inferred environment (e.g.,
AWS_BATCH_JOB_ID
environment variable set) - Job definition parameters
If no valid job definition exists (see AWSBatch.job_definition_arn
then a new job definition will be created and registered based on the job parameters.
BatchJob
AWSBatch.BatchJob
— TypeBatchJob
Stores a batch job id in order to:
describe
a job and its parameters- check on the
status
of a job wait
for a job to complete- fetch
log_events
Fields
id::AbstractString
: jobId
AWSBatch.submit
— Functionsubmit(
name::AbstractString,
definition::JobDefinition,
queue::JobQueue;
container::AbstractDict=Dict(),
parameters::Dict{String,String}=Dict{String, String}(),
num_jobs::Integer=1,
) -> BatchJob
Handles submitting the batch job. Returns a BatchJob
wrapper for the id.
AWSBatch.describe
— Methoddescribe(job::BatchJob) -> Dict
Provides details about the AWS batch job.
AWSBatch.JobDefinition
— MethodJobDefinition
Returns the job definition corresponding to a batch job.
AWSBatch.status
— Methodstatus(job::BatchJob) -> JobState
Returns the current status of a job.
Base.wait
— Methodwait(
cond::Function,
job::BatchJob;
timeout=600,
delay=5
)
Polls the batch job state until it hits one of the conditions in cond
. The loop will exit if it hits a failure
condition and will not catch any excpetions. The polling interval can be controlled with delay
and timeout
provides a maximum polling time.
Examples
julia> wait(state -> state < SUCCEEDED, job)
true
Base.wait
— Methodwait(
job::BatchJob,
cond::Vector{JobState}=[RUNNING, SUCCEEDED],
failure::Vector{JobState}=[FAILED];
kwargs...,
)
Polls the batch job state until it hits one of the conditions in cond
. The loop will exit if it hits a failure
condition and will not catch any excpetions. The polling interval can be controlled with delay
and timeout
provides a maximum polling time.
AWSBatch.log_events
— Methodlog_events(job::BatchJob) -> Union{Vector{LogEvent}, Nothing}
Fetches the logStreamName, fetches the CloudWatch logs, and returns a vector of log events. If the log stream does not currently exist then nothing
is returned.
NOTES:
- The
logStreamName
isn't available until the job is RUNNING, so you may want to usewait(job)
orwait(job, [AWSBatch.SUCCEEDED])
prior to calling this function.
JobDefinition
AWSBatch.JobDefinition
— TypeJobDefinition
Stores the job definition arn including the revision.
AWSBatch.ComputeEnvironment
— TypeComputeEnvironment
An object representing an AWS batch compute environment.
AWSBatch.create_compute_environment
— Functioncreate_compute_environment(name;
managed=true,
role="",
resources=Dict(),
enabled=true,
tags=Dict(),
aws_config=global_aws_config())
Create a compute environment of type type
with name name
.
See the AWS docs here.
AWSBatch.list_job_definitions
— Functionlist_job_definitions(;aws_config=global_aws_config())
Get a list of JobDefinition
objects via Batch.decsribe_job_definitions()
.
AWSBatch.job_definition_arn
— Methodjob_definition_arn(
definition_name::AbstractString;
image::AbstractString="",
role::AbstractString="",
aws_config::AbstractAWSConfig=global_aws_config(),
) -> Union{AbstractString, Nothing}
Looks up the ARN (Amazon Resource Name) for the latest job definition that can be reused. Returns a JobDefinition with the ARN that can be reused or nothing
.
A job definition can only be reused if:
- status = ACTIVE
- type = container
- image = the current job's image
- jobRoleArn = the current job's role
AWSBatch.register
— Methodregister(
definition_name::AbstractString;
role::AbstractString="",
image::AbstractString="",
vcpus::Integer=1,
memory::Integer=1024,
cmd::Cmd=``,
region::AbstractString="",
parameters::Dict{String,String}=Dict{String, String}(),
) -> JobDefinition
Registers a new job definition.
AWSBatch.deregister
— Methodderegister(job::JobDefinition)
Deregisters an AWS Batch job.
AWSBatch.isregistered
— Methodisregistered(definition::JobDefinition; aws_config=global_aws_config()) -> Bool
Checks if a JobDefinition is registered.
AWSBatch.describe
— Methoddescribe(definition::JobDefinition; aws_config=global_aws_config()) -> Dict
Describes a job definition as a dictionary. Requires the IAM permissions "batch:DescribeJobDefinitions".
JobQueue
AWSBatch.JobQueue
— TypeAWSBatch.create_job_queue
— Functioncreate_job_queue(name, envs, priority=1; aws_config=global_aws_config())
Create a job queue with name name
and priority priority
returning the associated JobQueue
object. envs
must be an iterator of compute environments given by ARN.
See the AWS docs here.
AWSBatch.list_job_queues
— Functionlist_job_queues(;aws_config=global_aws_config())
Get a list of JobQueue
objects as returned by Batch.describe_job_queues()
.
JobState
AWSBatch.JobState
— TypeLogEvent
AWSBatch.LogEvent
— TypeLogEvent
A struct for representing an event in an AWS Batch job log.