Edit on GitHub

S3 Interaction

AWSS3.s3_get — Function
s3_get([::AbstractAWSConfig], bucket, path; <keyword arguments>)

Retrieves an object from the bucket for a given path.

Optional Arguments

  • version=nothing: version of object to get.
  • retry=true: try again on "NoSuchBucket", "NoSuchKey" (common if object was recently created).
  • raw=false: return response as Vector{UInt8}
  • byte_range=nothing: given an iterator of (start_byte, end_byte) gets only the range of bytes of the object from start_byte to end_byte. For example, byte_range=1:4 gets bytes 1 to 4 inclusive. Arguments should use the Julia convention of 1-based indexing.
  • header::Dict{String,String}: pass in an HTTP header to the request.

As an example of how to set custom HTTP headers, the below is equivalent to s3_get(aws, bucket, path; byte_range=range):

s3_get(aws, bucket, path; headers=Dict{String,String}("Range" => "bytes=$(first(range)-1)-$(last(range)-1)"))

API Calls

Permissions

  • s3:GetObject: (conditional): required when version === nothing.
  • s3:GetObjectVersion: (conditional): required when version !== nothing.
  • s3:ListBucket (optional): allows requests to non-existent objects to throw a exception with HTTP status code 404 (Not Found) instead of HTTP status code 403 (Access Denied).
source
AWSS3.s3_get_file — Function
s3_get_file([::AbstractAWSConfig], bucket, path, filename; [version=], kwargs...)

Like s3_get but streams result directly to filename. Keyword arguments accept are the same as those for s3_get.

source
AWSS3.s3_get_meta — Function

s3getmeta([::AbstractAWSConfig], bucket, path; [version], kwargs...)

Retrieves metadata from an object without returning the object itself.

API Calls

Permissions

  • s3:GetObject (conditional): required when version === nothing.
  • s3:GetObjectVersion: (conditional): required when version !== nothing.
  • s3:ListBucket (optional): allows requests to non-existent objects to throw a exception with HTTP status code 404 (Not Found) instead of HTTP status code 403 (Access Denied).
source
AWSS3.s3_exists — Function
s3_exists([::AbstractAWSConfig], bucket, path; version=nothing)

Returns if an object exists with the key path in the bucket. If a version is specified then an object must exist with the specified version identifier.

Note that the AWS API's support for object versioning is quite limited and this check will involve try/catch logic if a version is specified.

API Calls

Permissions

  • s3:GetObject (conditional): required when version === nothing.
  • s3:GetObjectVersion: (conditional): required when version !== nothing.
  • s3:ListBucket (optional): allows requests to non-existent objects to throw a exception with HTTP status code 404 (Not Found) instead of HTTP status code 403 (Access Denied).
source
AWSS3.s3_delete — Function
s3_delete([::AbstractAWSConfig], bucket, path; [version], kwargs...)

Deletes an object from a bucket. The version argument can be used to delete a specific version.

API Calls

Permissions

source
AWSS3.s3_copy — Function
s3_copy([::AbstractAWSConfig], bucket, path; acl::AbstractString="",
        to_bucket=bucket, to_path=path, metadata::AbstractDict=SSDict(),
        parse_response::Bool=true, kwargs...)

Copy the object at path in bucket to to_path in to_bucket.

Optional Arguments

  • acl=; x-amz-acl header for setting access permissions with canned config. See here.
  • metadata::Dict=; x-amz-meta- headers.
  • parse_response::Bool=; when false, return raw AWS.Response
  • kwargs; additional kwargs passed through into S3.copy_object

API Calls

Permissions

source
AWSS3.s3_create_bucket — Function
s3_create_bucket([::AbstractAWSConfig], bucket; kwargs...)

Creates a new S3 bucket with the globally unique bucket name. The bucket will be created AWS region associated with the AbstractAWSConfig (defaults to "us-east-1").

API Calls

Permissions

source
AWSS3.s3_put_cors — Function
s3_put_cors([::AbstractAWSConfig], bucket, cors_config; kwargs...)

PUT Bucket cors

s3_put_cors("my_bucket", """
    <?xml version="1.0" encoding="UTF-8"?>
    <CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
        <CORSRule>
            <AllowedOrigin>http://my.domain.com</AllowedOrigin>
            <AllowedOrigin>http://my.other.domain.com</AllowedOrigin>
            <AllowedMethod>GET</AllowedMethod>
            <AllowedMethod>HEAD</AllowedMethod>
            <AllowedHeader>*</AllowedHeader>
            <ExposeHeader>Content-Range</ExposeHeader>
        </CORSRule>
    </CORSConfiguration>
"""
source
AWSS3.s3_put_tags — Function
s3_put_tags([::AbstractAWSConfig], bucket, [path], tags::Dict; kwargs...)

Sets the tags for a bucket or an existing object. When path is specified then tagging is performed on the object, otherwise it is performed on the bucket.

See also s3_put_tags, s3_delete_tags, and s3_put's tag option.

API Calls

Permissions

source
AWSS3.s3_get_tags — Function
s3_get_tags([::AbstractAWSConfig], bucket, [path]; kwargs...)

Get the tags associated with a bucket or an existing object. When path is specified then tag retrieval is performed on the object, otherwise it is performed on the bucket.

See also s3_put_tags and s3_delete_tags.

API Calls

Permissions

source
AWSS3.s3_delete_tags — Function
s3_delete_tags([::AbstractAWSConfig], bucket, [path])

Delete the tags associated with a bucket or an existing object. When path is specified then tag deletion is performed on the object, otherwise it is performed on the bucket.

See also s3_put_tags and s3_get_tags.

API Calls

Permissions

source
AWSS3.s3_list_objects — Function
s3_list_objects([::AbstractAWSConfig], bucket, [path_prefix]; delimiter="/", max_items=1000, kwargs...)

List Objects in bucket with optional path_prefix.

Returns an iterator of Dicts with keys Key, LastModified, ETag, Size, Owner, StorageClass.

source
AWSS3.s3_purge_versions — Function
s3_purge_versions([::AbstractAWSConfig], bucket, [path_prefix [, pattern]]; kwargs...)

Removes all versions of an object except for the latest version. When path_prefix is provided then only objects whose key starts with path_prefix will be purged. Use of pattern further restricts which objects are purged by only purging object keys containing the pattern (i.e string literal or regex). When both path_prefix and pattern are not' specified then all objects in the bucket will be purged.

API Calls

Permissions

source
AWSS3.s3_put — Function
s3_put([::AbstractAWSConfig], bucket, path, data, data_type="", encoding="";
       acl::AbstractString="", metadata::SSDict=SSDict(), tags::AbstractDict=SSDict(),
       parse_response::Bool=true, kwargs...)

PUT Object data at path in bucket.

Optional Arguments

  • data_type=; Content-Type header.
  • encoding=; Content-Encoding header.
  • acl=; x-amz-acl header for setting access permissions with canned config. See here.
  • metadata::Dict=; x-amz-meta- headers.
  • tags::Dict=; x-amz-tagging- headers (see also s3_put_tags and s3_get_tags).
  • parse_response::Bool=; when false, return raw AWS.Response
  • kwargs; additional kwargs passed through into S3.put_object
source
AWSS3.s3_sign_url — Function
s3_sign_url([::AbstractAWSConfig], bucket, path, [seconds=3600];
            [verb="GET"], [content_type="application/octet-stream"],
            [protocol="http"], [signature_version="v4"])

Create a pre-signed url for bucket and path (expires after for seconds).

To create an upload URL use verb="PUT" and set content_type to match the type used in the Content-Type header of the PUT request.

For compatibility, the signature version 2 signing process can be used by setting signature_version="v2" but it is recommended that the default version 4 is used.

url = s3_sign_url("my_bucket", "my_file.txt"; verb="PUT")
Requests.put(URI(url), "Hello!")
url = s3_sign_url("my_bucket", "my_file.txt";
                  verb="PUT", content_type="text/plain")

Requests.put(URI(url), "Hello!";
             headers=Dict("Content-Type" => "text/plain"))

Permissions

  • s3:GetObject (conditional): required permission when verb="GET".
  • s3:PutObject (conditional): required permission when verb="PUT".
source
AWSS3.s3_nuke_bucket — Function
s3_nuke_bucket([::AbstractAWSConfig], bucket_name)

Deletes a bucket including all of the object versions in that bucket. Users should not call this function unless they are certain they want to permanently delete all of the data that resides within this bucket.

The s3_nuke_bucket is purposefully not exported as a safe guard against accidental usage.

Warning

Permanent data loss will occur when using this function. Do not use this function unless you understand the risks. By using this function you accept all responsibility around any repercussions with the loss of this data.

API Calls

Permissions

source

S3Path

Note that S3Path implements the AbstractPath interface, some the FilePathsBase documentation for the interface here.

AWSS3.S3Path — Type
S3Path()
S3Path(str::AbstractString; version::Union{Nothing, AbstractString}=nothing, config::Union{Nothing, AWS.AbstractAWSConfig}=nothing)
S3Path(path::S3Path; isdirectory=path.isdirectory, version=path.version, config=path.config)

Construct a new AWS S3 path type which should be of the form "s3://<bucket>/prefix/to/my/object".

NOTES:

  • Directories are required to have a trailing / due to how S3 distinguishes files from folders, as internally they're just keys to objects.
  • Objects p"s3://bucket/a" and p"s3://bucket/a/b" can co-exist. If both of these objects exist listing the keys for p"s3://bucket/a" returns [p"s3://bucket/a"] while p"s3://bucket/a/" returns [p"s3://bucket/a/b"].
  • The drive property will return "s3://<bucket>"
  • On top of the standard path properties (e.g., segments, root, drive, separator), S3Paths also support bucket and key properties for your convenience.
  • If version argument is nothing, will return latest version of object. Version can be provided via either kwarg version or as suffix "?versionId=<object_version>" of str, e.g., "s3://<bucket>/prefix/to/my/object?versionId=<object_version>".
  • If config is left at its default value of nothing, then the latest global_aws_config() will be used in any operations involving the path. To "freeze" the config at construction time, explicitly pass an AbstractAWSConfig to the config keyword argument.
source
Base.stat — Function
stat(fp::S3Path)

Return the status struct for the S3 path analogously to stat for local directories.

Note that this cannot be used on a directory. This is because S3 is a pure key-value store and internally does not have a concept of directories. In some cases, a directory may actually be an empty file, in which case you should use s3_get_meta.

source
Base.Filesystem.mkdir — Function
mkdir(fp::S3Path; recursive=false, exist_ok=false)

Create an empty directory within an existing bucket for the S3 path fp. If recursive, this will create any previously non-existent directories which would contain fp. An error will be thrown if an object exists at fp unless exist_ok.

Note

Creating a directory in S3 creates a 0-byte object with a key set to the provided directory name.

source
Base.read — Function
read(fp::S3Path; byte_range=nothing)

Fetch data from the S3 path as a Vector{UInt8}. A subset of the object can be specified with byte_range which should be a contiguous integer range, e.g. 1:4.

source
AWSS3.get_config — Function
get_config(fp::S3Path)

Returns the AWS configuration used by the path fp. This can be stored within the path itself, but if not it will be fetched with AWS.global_aws_config().

source

Internal

AWSS3._s3_exists_dir — Function
_s3_exists_dir(aws::AbstractAWSConfig, bucket, path)

An internal function used by s3_exists.

Checks if the given directory exists within the bucket. Since S3 uses a flat structure, as opposed to being hierarchical like a file system, directories are actually just a collection of object keys which share a common prefix. S3 implements empty directories as 0-byte objects with keys ending with the delimiter.

It is possible to create non 0-byte objects with a key ending in the delimiter (e.g. s3_put(bucket, "abomination/", "If I cannot inspire love, I will cause fear!")) which the AWS console interprets as the directory "abmonination" containing the object "/".

source
AWSS3.s3_exists_versioned — Function
s3_exists_versioned([::AbstractAWSConfig], bucket, path, version)

Returns if an object version exists with the key path in the bucket.

Note that the AWS API's support for object versioning is quite limited and this check will involve try/catch logic. Prefer using s3_exists_unversioned where possible for more performant checks.

See s3_exists and s3_exists_unversioned.

API Calls

Permissions

  • s3:GetObjectVersion
  • s3:ListBucket (optional): allows requests to non-existent objects to throw a exception with HTTP status code 404 (Not Found) instead of HTTP status code 403 (Access Denied).
source