AWSBatch

Docs: stable CI

AWSBatch.jl provides a small set of methods for working with AWS Batch jobs from julia.

Installation

AWSBatch assumes that you already have an AWS account configured with:

  1. An ECR repository and a docker image pushed to it [1].
  2. An IAM role to apply to the batch jobs.
  3. A compute environment and job queue for submitting jobs to [2].

Please review the "Getting Started with AWS Batch" guide and example CloudFormation template for more details.

Basic Usage

julia> using AWSBatch

julia> job = run_batch(
           name="Demo",
           definition="AWSBatchJobDefinition",
           queue="AWSBatchJobQueue",
           image = "000000000000.dkr.ecr.us-east-1.amazonaws.com/demo:latest",
           role = "arn:aws:iam::000000000000:role/AWSBatchJobRole",
           vcpus = 1,
           memory = 1024,
           cmd = `julia -e 'println("Hello World!")'`,
       )
AWSBatch.BatchJob("00000000-0000-0000-0000-000000000000")

julia> wait(job, [AWSBatch.SUCCEEDED])
true

julia> results = log_events(job)
1-element Array{AWSBatch.LogEvent,1}:
 AWSBatch.LogEvent("00000000000000000000000000000000000000000000000000000000", 2018-04-23T19:41:18.765, 2018-04-23T19:41:18.677, "Hello World!")

AWSBatch also supports Memento logging for more detailed usage information.

API

AWSBatch.run_batchMethod
run_batch(;
    name::AbstractString="",
    queue::AbstractString="",
    region::AbstractString="",
    definition::Union{AbstractString, JobDefinition, Nothing}=nothing,
    image::AbstractString="",
    vcpus::Integer=1,
    memory::Integer=-1,
    role::AbstractString="",
    cmd::Cmd=``,
    num_jobs::Integer=1,
    parameters::Dict{String, String}=Dict{String, String}(),
) -> BatchJob

Handles submitting a BatchJob based on various potential defaults. For example, default job fields can be inferred from an existing job definition or an existing job (if currently running in a batch job).

Order of priority from highest to lowest:

  1. Explict arguments passed in via kwargs.
  2. Inferred environment (e.g., AWS_BATCH_JOB_ID environment variable set)
  3. Job definition parameters

If no valid job definition exists (see AWSBatch.job_definition_arn then a new job definition will be created and registered based on the job parameters.

source

BatchJob

AWSBatch.BatchJobType
BatchJob

Stores a batch job id in order to:

  • describe a job and its parameters
  • check on the status of a job
  • wait for a job to complete
  • fetch log_events

Fields

  • id::AbstractString: jobId
source
AWSBatch.submitFunction
submit(
    name::AbstractString,
    definition::JobDefinition,
    queue::JobQueue;
    container::AbstractDict=Dict(),
    parameters::Dict{String,String}=Dict{String, String}(),
    num_jobs::Integer=1,
) -> BatchJob

Handles submitting the batch job. Returns a BatchJob wrapper for the id.

source
Base.waitMethod
wait(
    cond::Function,
    job::BatchJob;
    timeout=600,
    delay=5
)

Polls the batch job state until it hits one of the conditions in cond. The loop will exit if it hits a failure condition and will not catch any excpetions. The polling interval can be controlled with delay and timeout provides a maximum polling time.

Examples

julia> wait(state -> state < SUCCEEDED, job)
true
source
Base.waitMethod
wait(
    job::BatchJob,
    cond::Vector{JobState}=[RUNNING, SUCCEEDED],
    failure::Vector{JobState}=[FAILED];
    kwargs...,
)

Polls the batch job state until it hits one of the conditions in cond. The loop will exit if it hits a failure condition and will not catch any excpetions. The polling interval can be controlled with delay and timeout provides a maximum polling time.

source
AWSBatch.log_eventsMethod
log_events(job::BatchJob) -> Union{Vector{LogEvent}, Nothing}

Fetches the logStreamName, fetches the CloudWatch logs, and returns a vector of log events. If the log stream does not currently exist then nothing is returned.

NOTES:

  • The logStreamName isn't available until the job is RUNNING, so you may want to use wait(job) or wait(job, [AWSBatch.SUCCEEDED]) prior to calling this function.
source

JobDefinition

AWSBatch.create_compute_environmentFunction
create_compute_environment(name;
                           managed=true,
                           role="",
                           resources=Dict(),
                           enabled=true,
                           tags=Dict(),
                           aws_config=global_aws_config())

Create a compute environment of type type with name name.

See the AWS docs here.

source
AWSBatch.job_definition_arnMethod
job_definition_arn(
    definition_name::AbstractString;
    image::AbstractString="",
    role::AbstractString="",
    aws_config::AbstractAWSConfig=global_aws_config(),
) -> Union{AbstractString, Nothing}

Looks up the ARN (Amazon Resource Name) for the latest job definition that can be reused. Returns a JobDefinition with the ARN that can be reused or nothing.

A job definition can only be reused if:

  1. status = ACTIVE
  2. type = container
  3. image = the current job's image
  4. jobRoleArn = the current job's role
source
AWSBatch.registerMethod
register(
    definition_name::AbstractString;
    role::AbstractString="",
    image::AbstractString="",
    vcpus::Integer=1,
    memory::Integer=1024,
    cmd::Cmd=``,
    region::AbstractString="",
    parameters::Dict{String,String}=Dict{String, String}(),
) -> JobDefinition

Registers a new job definition.

source
AWSBatch.isregisteredMethod
isregistered(definition::JobDefinition; aws_config=global_aws_config()) -> Bool

Checks if a JobDefinition is registered.

source
AWSBatch.describeMethod
describe(definition::JobDefinition; aws_config=global_aws_config()) -> Dict

Describes a job definition as a dictionary. Requires the IAM permissions "batch:DescribeJobDefinitions".

source

JobQueue

AWSBatch.create_job_queueFunction
create_job_queue(name, envs, priority=1; aws_config=global_aws_config())

Create a job queue with name name and priority priority returning the associated JobQueue object. envs must be an iterator of compute environments given by ARN.

See the AWS docs here.

source
AWSBatch.list_job_queuesFunction
list_job_queues(;aws_config=global_aws_config())

Get a list of JobQueue objects as returned by Batch.describe_job_queues().

source

JobState

LogEvent