S3 Guides

Guides for using awskit-s3 with AWS S3 object and bucket workflows. The client surface is streaming-first, endpoint-configurable, and result-returning.

Object Operations

Primitive object operations live under Object. They use runtime-owned request bodies and scoped response body readers.

Upload

let body = Awskit_s3_eio.Runtime.Request_body.of_string "hello" in
Awskit_s3_eio.Object.put s3
  ~bucket:"my-bucket"
  ~key:"hello.txt"
  ~body
  ()

Primitive uploads require a known content length. The runtime records length and payload hash in its request-body descriptor before signing. Unknown-length SigV4 aws-chunked streaming is not implemented today.

For custom Runtime.Request_body.of_stream values, declare the exact content_length and emit exactly that number of bytes. Producer callback errors are returned as request body failures. Prefer Runtime.Request_body.of_string or Runtime.Request_body.of_bytes when the body is already buffered.

Options carry object metadata, content type, storage class, tags, preconditions, checksum headers, and server-side encryption headers:

let options =
  {
    Awskit_s3.Put_object.default_options with
    content_type = Some "text/plain";
    metadata = [ ("origin", "example") ];
    tags = [ { Awskit_s3.Tag.key = "env"; value = "dev" } ];
  }
in
Awskit_s3_eio.Object.put s3
  ~bucket:"my-bucket"
  ~key:"hello.txt"
  ~options
  ~body:(Awskit_s3_eio.Runtime.Request_body.of_string "hello")
  ()

Checksums

Object.Checksum.value carries an explicit precomputed checksum value. Use it for PutObject, UploadPart, and CompleteMultipartUpload value headers. CopyObject and CreateMultipartUpload use Object.Checksum.Algorithm.t, while GetObject and HeadObject use Object.Checksum.Mode.Enabled.

Runtime-backed operations parse all modeled checksum response headers plus optional checksum type metadata from PutObject, GetObject, HeadObject, UploadPart, and CompleteMultipartUpload. Presigned PUT URLs sign explicit checksum value headers when configured.

Preconditions

Object write, read, delete, and copy operations expose S3 conditional request records under Object.Preconditions. Runtime-backed clients map those records to the AWS headers for If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since, copy-source preconditions, and S3 conditional delete headers.

Download

Object.get scopes the response body to the consume callback. Do not store the reader outside that callback. Buffering helpers such as Object.get_as_string ~max_bytes keep response-size limits explicit.

let consume reader =
  let bytes = Bytes.create 8192 in
  match Awskit_s3_eio.Runtime.Response_body.read reader bytes ~off:0 ~len:8192 with
  | Error _ as error -> error
  | Ok n -> Ok (Bytes.sub_string bytes 0 n)
in
Awskit_s3_eio.Object.get s3
  ~bucket:"my-bucket"
  ~key:"large.bin"
  ~consume
  ()

Use Object.head when only metadata is needed. Use Object.exists for boolean existence checks. Missing-object and missing-bucket behavior is reported as structured service errors by primitive operations.

Copy, Delete, List, Tagging

The object surface also includes:

Operation pagination helpers preserve the regular List_objects_v2.options record and update only the continuation token between requests. Use max_pages to cap traversal:

let keys =
  Awskit_s3_eio.Object.List_objects_v2.keys s3
    ~bucket:"my-bucket"
    ~options:{ Awskit_s3.List_objects_v2.default_options with prefix = Some "logs/" }
    ~max_pages:10
    ()

Versioning

Bucket.Versioning.put enables bucket versioning. Object writes, copies, and multipart completion return version IDs when the bucket is versioned. Reads, heads, and deletes accept a version_id option. Copy requests can select a source version with Copy_object.source_version_id.

Deleting the current object in a versioned bucket creates a delete marker. Object.list_versions returns object versions and delete markers, and Object.List_object_versions provides pagination helpers for traversing the full history. List_object_versions is the public alias for version-listing options and page types:

let options =
  {
    Awskit_s3.List_object_versions.default_options with
    prefix = Some "logs/";
  }
in
Awskit_s3_eio.Object.List_object_versions.object_versions s3
  ~bucket:"my-bucket"
  ~options
  ()

In-Memory Object Helpers

Object.put_string, Object.put_bytes, Object.get_as_string, and Object.get_as_bytes are for explicit in-memory use. Download helpers require max_bytes to avoid accidental unbounded buffering.

Awskit_s3_eio.Object.put_string s3
  ~bucket:"my-bucket"
  ~key:"hello.txt"
  "hello"
Awskit_s3_eio.Object.get_as_string s3
  ~bucket:"my-bucket"
  ~key:"hello.txt"
  ~max_bytes:1_048_576L
  ()

If the response body exceeds max_bytes, the helper returns Error (Awskit.Error.Body _).

Object.Transfer

Unix-capable adapters expose local-path helpers under Object.Transfer. Keeping these helpers at the adapter layer lets the pure awskit-s3 core stay independent of filesystem APIs.

put_file and get_file are one-request PutObject and GetObject helpers. upload_file and download_file choose an optimized strategy using Awskit_s3.Transfer options: small transfers use one request, while larger transfers use multipart upload or ranged GetObject requests. multipart_upload_file and resume_multipart_upload_file are explicit multipart upload helpers. Filesystem transfer helpers live only in the Eio and Lwt Unix adapters; they are not part of the pure awskit-s3 core.

Eio

Awskit_s3_eio.Object.Transfer.upload_file s3
  ~bucket:"my-bucket"
  ~key:"archive.tar"
  ~on_progress:(fun bytes -> Fmt.pr "uploaded %Ld bytes@." bytes)
  ~path
  ()
Awskit_s3_eio.Object.Transfer.put_file s3
  ~bucket:"my-bucket"
  ~key:"single-request-upload.tar"
  ~path
  ()
Awskit_s3_eio.Object.Transfer.download_file s3
  ~bucket:"my-bucket"
  ~key:"archive.tar"
  ~on_progress:(fun bytes -> Fmt.pr "downloaded %Ld bytes@." bytes)
  ~path
  ()
Awskit_s3_eio.Object.Transfer.get_file s3
  ~bucket:"my-bucket"
  ~key:"single-request-download.tar"
  ~path
  ()

Lwt Unix

let* result =
  Awskit_s3_lwt_unix.Object.Transfer.upload_file s3
    ~bucket:"my-bucket"
    ~key:"archive.tar"
    ~path:"/tmp/archive.tar"
    ()

For explicit multipart upload, use multipart_upload_file. It uploads file parts with bounded concurrency from Transfer.upload_options and aborts the multipart upload if a fresh upload fails:

let options =
  { Awskit_s3.Transfer.default_upload_options with concurrency = 4 }
in
let* result =
  Awskit_s3_lwt_unix.Object.Transfer.multipart_upload_file s3
    ~bucket:"my-bucket"
    ~key:"large.tar"
    ~options
    ~path:"/tmp/large.tar"
    ~on_progress:(fun bytes -> Fmt.pr "uploaded %Ld bytes@." bytes)
    ()

To resume an interrupted multipart upload, keep the upload_id. The resume helper lists uploaded parts, skips matching part numbers and sizes, uploads the missing parts, and completes the upload. Existing uploads are left open on failure, so the same upload ID can be retried:

let* result =
  Awskit_s3_lwt_unix.Object.Transfer.resume_multipart_upload_file s3
    ~bucket:"my-bucket"
    ~key:"large.tar"
    ~upload_id
    ~path:"/tmp/large.tar"
    ()

AWS S3 Endpoint Configuration

S3 endpoint configuration is part of client construction. The defaults use the standard regional AWS S3 endpoint for the selected region.

let s3 =
  Awskit_s3_eio.create
    ~sw
    ~env
    ~region
    ~credentials
    ~endpoint_variant:`Dualstack
    ~addressing_style:`Auto
    ()

Pass endpoint for S3 endpoints that are not the default AWS regional host, including local test services:

let endpoint =
  Awskit.Endpoint.of_string_exn "http://127.0.0.1:9000"
in
let s3 =
  Awskit_s3_eio.create
    ~sw
    ~env
    ~region
    ~credentials
    ~endpoint
    ~addressing_style:`Path
    ()

Endpoint variants:

Addressing styles:

Transfer Acceleration requires virtual-hosted-style-compatible bucket names and must be enabled on the bucket in AWS.

Bucket Operations

Bucket covers general-purpose bucket basics and the first configuration subresources:

Bucket.Policy sends and receives validated JSON policy payloads for bucket access configuration. Applications can construct those documents with their own policy model and pass the serialized JSON to the client.

Retained bucket operations that target an existing bucket accept expected_bucket_owner. Awskit sends it as x-amz-expected-bucket-owner so AWS can reject calls that resolve to a bucket owned by a different account.

let encryption =
  {
    Awskit_s3.Bucket.Encryption.rules =
      [
        {
          Awskit_s3.Bucket.Encryption.Rule.sse_algorithm =
            Some Awskit_s3.Bucket.Encryption.Algorithm.Aes256;
          kms_master_key_id = None;
          bucket_key_enabled = None;
          blocked_encryption_types = [];
        };
      ];
  }
in
Awskit_s3_eio.Bucket.Encryption.put s3
  ~bucket:"my-bucket"
  ~expected_bucket_owner:"123456789012"
  encryption

Multipart Uploads

Multipart exposes the primitive AWS multipart flow:

  1. Multipart.create_upload starts an upload and returns an upload ID.
  2. Multipart.upload_part uploads a known-length part.
  3. Multipart.complete_upload completes the upload with ordered parts.
  4. Multipart.abort_upload cancels an upload.
  5. Multipart.list_parts inspects one page of uploaded parts.
  6. Multipart.List_parts.pages and parts follow ListParts part-number markers.

Multipart.upload_part follows the same request body contract as Object.put: parts require a known exact length, and custom stream producer errors are part of the upload result.

Presigned URLs

Use adapter-level Presigned modules when a connection already exists, or use standalone Awskit_s3.Presigned helpers when generating URLs without an adapter connection.

Awskit_s3_eio.Presigned.get_object s3
  ~bucket:"my-bucket"
  ~key:"public.txt"
  ()
Awskit_s3.Presigned.put_object
  ~region
  ~credentials
  ~now
  ~endpoint
  ~addressing_style:`Path
  ~bucket:"my-bucket"
  ~key:"upload.txt"
  ()

Multipart direct-upload flows can presign individual UploadPart requests after Multipart.create_upload has produced an upload ID:

Awskit_s3.Presigned.upload_part
  ~region
  ~credentials
  ~now
  ~bucket:"my-bucket"
  ~key:"large.bin"
  ~upload_id
  ~part_number:1
  ()

Presigned URLs use UNSIGNED-PAYLOAD and the same endpoint/addressing options as normal operations. Returned result.signed_headers are the non-host headers that the caller must send with the URL. Use extra_signed_headers only for additional headers that must be included in the signature. Presigned PUT and UploadPart helpers sign explicit checksum value headers when configured, and object presigning options support expected_bucket_owner guard rails.

A typical direct-upload multipart flow is:

  1. Start the upload with Multipart.create_upload.
  2. Presign one UploadPart URL per part number with Presigned.upload_part.
  3. The uploader sends each part to the corresponding URL.
  4. Complete the upload with the returned part ETags.

Credentials

Eio S3 connections require explicit region and credentials.

let credentials =
  Awskit.Credentials.create_exn
    ~access_key_id:"AKIAIOSFODNN7EXAMPLE"
    ~secret_access_key:"wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
    ()
in
let region = Awskit.Region.of_string_exn "us-east-1" in
Awskit_s3_eio.create ~sw ~env ~region ~credentials ()

awskit-s3-lwt-unix can use explicit values or load the standard environment variables through awskit-lwt-unix:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN
AWS_REGION
AWS_DEFAULT_REGION
AWS_PROFILE
AWS_SHARED_CREDENTIALS_FILE
AWS_CONFIG_FILE
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
AWS_CONTAINER_CREDENTIALS_FULL_URI
AWS_CONTAINER_AUTHORIZATION_TOKEN
AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE
Awskit_s3_lwt_unix.create ()

The Unix adapters cover static credentials, environment variables, shared AWS config files, named static profiles, ECS container credentials, and EC2 instance profile credentials. Metadata credentials are resolved through the Lwt-Unix runtime credential source and refreshed before expiration.

Errors

S3 operations return (value, Awskit_s3.Error.t) result or that result inside the selected runtime monad.

Awskit_s3.Error.t is Awskit.Error.t. It preserves validation, signing, transport, service, decode, and body-consumption failures.

Use S3 classifiers for service-specific checks:

match result with
| Ok value -> use value
| Error err when Awskit_s3.Error.is_no_such_key err -> handle_missing ()
| Error err when Awskit_s3.Error.is_precondition_failed err -> retry_later ()
| Error err -> log_error err

For service failures, Awskit_s3.Error.service_code exposes the AWS error code when present.

Retries

Runtime-backed S3 clients use Awskit.Retry.default unless a runtime adapter receives a custom retry_policy. Retries are centralized around the signed S3 request, so retryable service responses are classified before being returned to callers.

Only replayable request bodies are retried. In-memory bodies created with Runtime.Request_body.of_string and Runtime.Request_body.of_bytes are replayable. Custom stream bodies are retried only when their Awskit.Body.Request.descriptor marks them replayable. Do not mark a stream replayable unless retrying can recreate the same bytes from the beginning.

Configure retry attempts, capped exponential backoff, and jitter explicitly:

let retry_policy =
  Awskit.Retry.create_exn
    ~max_attempts:4
    ~base_delay:(Ptime.Span.of_float_s 0.2 |> Option.get)
    ~max_delay:(Ptime.Span.of_float_s 3.0 |> Option.get)
    ~jitter:0.5
    ()
in
Awskit_s3_lwt_unix.create ~retry_policy ()

jitter is in the range [0, 1]. The default 0 keeps deterministic delays. Values greater than zero spread retry waits below the capped exponential delay.

Local MinIO Contract Testing

Normal tests do not require a network service. For local wire-level checks against MinIO, start the included Compose stack and run the opt-in alias:

docker compose up -d
dune build @minio-contract

The contract runner defaults to http://127.0.0.1:9000 with the minioadmin/minioadmin credentials from docker-compose.yml. Override with AWSKIT_S3_MINIO_ENDPOINT, AWSKIT_S3_MINIO_ACCESS_KEY_ID, AWSKIT_S3_MINIO_SECRET_ACCESS_KEY, and AWSKIT_S3_MINIO_REGION.