Amazon S3 REST API with curl

Lately I’ve been more and more attracted to the AWS platform. My current pet project is a simple OS X screenshot sharing app. S3 instantly felt like the appropriate storage for the files.

At the moment, there is no official AWS SDK for Mac. In a way this makes my task more interesting. I can dive deeper into the AWS REST API and I can exercise my Swift-fu in a challenging task.

The major challenge is performing a successful, authenticated S3 REST API request. The new Signature Version 4 signing process requires jumping through a few hoops to sign the request correctly.

The AWS documentation goes deep and explains all the steps, but to me, nothing beats seeing an actual, successful request in the terminal. Actually getting a 200 OK with curl took me two evenings of trial and error.

Here are my notes. Perhaps you will find them useful.

Why?

Why even bother with this? Who cares about the REST API? Wouldn’t a sane person just use the official SDKs and never get down to the REST level? Plus, there’s aws-cli if you want to do things in the terminal.

Perhaps. However, as was my case, an official SDK for your platform or programming language might not be available. Furthermore, the challenge to do everything by hand allowed me to understand AWS a tiny bit better and was a fun exercise in command line, curl, and HTTP.

What we need

For this exercise we will need:

  • curl
  • a way to calculate a SHA256 digest of a string
  • a way to calculate a HMAC-SHA256 mac of a string (with both a string key and a binary/hexadecimal key)
  • a text editor (make sure you can omit trailing newline in a file)

I’m working on OSX 10.10 and I will be using the following:

  • curl 7.43.0
  • openssl 1.0.2 (you need at least openssl 1.0.0 for the hexkey option)

The process

Our primary job is constructing the Authorization header. It will contain the signature for the request. Generating the signature is a step best understood in 5 distinct steps:

  1. Canonical request
  2. String to sign
  3. Generate user’s signing key
  4. Calculate signature
  5. Build the “Authorization” header

I will walk through the steps for the simplest possible S3 request, GET Bucket. It’s a good candidate because it carries no payload and only requires the basic headers to complete successfully.

Afterwards, I will leave a few notes on what is needed for more complex requests.

To simplify the steps, I will be constructing two text files, which contain the intermediate representations required for the signing process.

Overview

I created a bucket my-precious-bucket in “US Standard” region. I will be using the virtual-host based addressing scheme.

The request I will be performing is thus:

Method
GET
Host
https://my-precious-bucket.s3.amazonaws.com
Headers
Authorization - will contain the request signature
Host - will contain my-precious-bucket.s3.amazonaws.com
x-amz-content-sha256 - required by AWS, must be the SHA256 digest of the payload (see below)
x-amz-date - required by AWS, must contain the timestamp of the request; the accepted format is quite flexible, I’m using ISO8601 basic format.

Example data used for all the calculations:

AWS access key ID
AKIAIOSFODNN7EXAMPLE
AWS secret access key
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Timestamp
20150915T124500Z (Tue, 15 Sep 2015 12:45:00 GMT)

The request will carry no payload (i.e. the body will be empty). This means that wherever a “payload hash” is required, we will provide an SHA256 hash of an empty string. And that is a constant value of e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855. This concerns the x-amz-content-sha256 header as well.

Step 1: Canonical request

First we need to prepare what’s called the “Canonical request”. This is a textual representation of the request we’re performing. The blueprint of this string is explained in the AWS docs and looks like this:

<HTTPMethod>\n
<CanonicalURI>\n
<CanonicalQueryString>\n
<CanonicalHeaders>\n
<SignedHeaders>\n
<HashedPayload>

It is our job to fill in the placeholders as required and describe the request we’re about to perform.

For GET Bucket, the result is as follows:

get-my-precious-bucket.creq
GET
/

host:my-precious-bucket.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20150915T124500Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Notes:

  • <CanonicalQueryString> is an empty string. Note however that the line must not be omitted and the newline is required.
  • <CanonicalHeaders> contain the HTTP headers we will be putting in the request, one per line. The names of the headers are lowercased and whitespace is stripped. In this very basic case, the only required headers are Host, x-amz-content-sha256 and x-amz-date.
  • <SignedHeaders> is a semicolon-separated list of the headers. We just repeat the ones we used for <CanonicalHeaders>
  • <HashedPayload> is the SHA256 checksum of the request payload. Since we carry no payload in this request, we append the SHA256 hash of an empty string.

I type this into a plain text file as get-my-precious-bucket.creq. Note there is no newline character after the last line. We will be using this file in the following step.

Step 2: String to sign

The actual signature will be calculated for a particular string, called “string to sign”. The required format of this string looks like this:

AWS4-HMAC-SHA256\n
<Timestamp>\n
<Scope>\n
<CanonicalRequestHash>

Again, I will use a text editor to compose this string as a four-line text file. I will be calling it get-my-precious-bucket.sts:

get-my-precious-bucket.sts
AWS4-HMAC-SHA256
20150915T124500Z
20150915/us-east-1/s3/aws4_request
ef7c45cc2b0f100ea5d65024643f5cbaf83e7ba2717108905acd605cfe17bc6b

Notes:

  • AWS4-HMAC-SHA256 is a constant string which needs to be placed in the first line. It specifies the hash algorithm used for request signing.
  • <Timestamp> is the current UTC time in ISO 8601 basic format.
  • <Scope> binds the request to a certain date, AWS region, and service. It must be in the format: <yyyyMMdd>/<Region>/<Service>/aws4_request, where aws4_request is a constant string.
  • <CanonicalRequestHash> is a hexadecimal representation of the SHA256 hash of our canonical request - the string from Step 1.

The <CanonicalRequestHash> can be obtained with openssl:

$ openssl dgst -sha256 get-my-precious-bucket.creq
SHA256(get-my-precious-bucket.creq)= ef7c45cc2b0f100ea5d65024643f5cbaf83e7ba2717108905acd605cfe17bc6b

Again, remember - no newline at the end of the file.

Step 3: Generate user’s signing key

We have the string to sign, now we need the key which we’ll use to sign it. The signing key is derived from the user’s secret access key, through a four-step HMAC-SHA256 hashing process.

AWS explains the process with examples in a number of programming languages. I will just stick to bash and calculate the signing key using the openssl dgst command.

Here’s a simple script signing_key.sh to perform the calculation:

signing_key.sh
#!/bin/bash
function hmac_sha256 {
  key="$1"
  data="$2"
  echo -n "$data" | openssl dgst -sha256 -mac HMAC -macopt "$key" | sed 's/^.* //'
}

secret="$1"
date="$2"
region="$3"
service="$4"

# Four-step signing key calculation
dateKey=$(hmac_sha256 key:"AWS4$secret" $date)
dateRegionKey=$(hmac_sha256 hexkey:$dateKey $region)
dateRegionServiceKey=$(hmac_sha256 hexkey:$dateRegionKey $service)
signingKey=$(hmac_sha256 hexkey:$dateRegionServiceKey "aws4_request")

echo $signingKey

To run the script, provide your AWS secret access key, the date, region and service name. Example:

$ ./signing_key.sh wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY 20150915 us-east-1 s3
7b0b3063e375aa1e25890e0cae1c674785b8d8709cd2bf11ec670b96587650da

The output is a hexadecimal representation of the signing key. Hold on to it for the next step.

Step 4: Calculate signature

We have the string to sign and we have the signing key - now we need one final HMAC-SHA256 calculation to obtain the signature. I will use openssl dgst one last time:

$ openssl dgst -sha256 \
               -mac HMAC \
               -macopt hexkey:7b0b3063e375aa1e25890e0cae1c674785b8d8709cd2bf11ec670b96587650da \
               get-my-precious-bucket.sts
HMAC-SHA256(get-my-precious-bucket.sts)= 182072eb53d85c36b2d791a1fa46a12d23454ec1e921b02075c23aee40166d5a

The hash 182072eb... is the request signature. Hold on to it.

Step 5: Build the “Authorization” header

The calculated signature needs to be placed in the Authorization header and included in the HTTP request. The format of the header is as follows:

Authorization: <Algorithm> Credential=<Access Key ID/Scope>, SignedHeaders=<SignedHeaders>, Signature=<Signature>

We’ve already seen most of the values in previous steps. The only addition is the <Access Key ID>, where we place the AWS access key ID. In our example, the complete header looks like this:

Authorization: AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20150915/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=182072eb53d85c36b2d791a1fa46a12d23454ec1e921b02075c23aee40166d5a

Final step: Perform the request

We’re ready to curl the request. We need to include the Authorization header, as well as the two required x-amz-content-sha256 and x-amz-date headers. We also need the Host header, but curl will place this one for us automatically, based on the URL provided.

Here’s the complete call:

curl -v https://my-precious-bucket.s3.amazonaws.com/ \
     -H "Authorization: AWS4-HMAC-SHA256 \
         Credential=AKIAIOSFODNN7EXAMPLE/20150915/us-east-1/s3/aws4_request, \
         SignedHeaders=host;x-amz-content-sha256;x-amz-date, \
         Signature=182072eb53d85c36b2d791a1fa46a12d23454ec1e921b02075c23aee40166d5a" \
     -H "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
     -H "x-amz-date: 20150915T124500Z"

If we’ve done everything right, we’ll receive the following (formatted for clarity):

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>my-precious-bucket</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>my-precious-file-1.txt</Key>
    <LastModified>2015-09-15T12:49:35.000Z</LastModified>
    <ETag>&quot;e243bb39c844b3543a7726576c869caf&quot;</ETag>
    <Size>7</Size>
    <Owner>
      <ID>948b6b7d545aaa9ed10fa52fcf06827960c607d47c96574101e3a68045fc872d</ID>
      <DisplayName>lukasz</DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
</ListBucketResult>

Moving further

If you’ve made it this far, congratulations! Correctly signing an AWS request is not trivial and there’s a reason why we use SDKs to do it for us.

The steps described above are the same for signing all authenticated S3 REST API requests. The simple example makes it easier to understand, but the process is the same throughout the API.

For more complex requests (e.g. PUT Object), keep the following in mind:

  • Your request might have a nonempty body (e.g. the file to upload), so the value for x-amz-content-sha256 and the <HashedPayload> line will be based on that. You can use openssl dgst -sha256 picture.jpg to get the digest of picture.jpg
  • If you include other headers in the request (e.g. Content-Type for PUT Object), you also need to include them (in lowercase!) in the Canonical Request representation.

Troubleshooting

If you’re having trouble, first check the following:

  • Ensure you’re not including whitespace where it doesn’t belong (or the other way around)
  • Use the appropriate key in the appropriate place
    • AWS access key ID goes into the Credential part of the Authorization header
    • AWS secret access key is used to generate your signing key
  • Your request must be performed within 15 minutes of the specified timestamp. Otherwise it will be rejected by the server.

Fortunately, error messages from the API are quite descriptive. For instance, if you fail to meet the 15 minutes time limit, you will receive a RequestTimeTooSkewed error.

If you have further problems, let me know in the comments. I will do my best to help.

Łukasz Adamczak's Picture

About Łukasz Adamczak

I'm a mobile & web developer based in Warsaw, Poland. On this blog I write down notes from my programming journeys.

Warsaw, Poland http://czak.pl

Comments