Łukasz Adamczak

Łukasz Adamczak
One developer's journeys through code

Amazon S3 REST API with curl

Sep 15, 2015
  • AWS

Lately I’ve been more and more attracted to the AWS platform. My current pet project is a simple OS X screenshot sharing app. S3 instantly felt like the appropriate storage for the files.

At the moment, there is no official AWS SDK for Mac. In a way this makes my task more interesting. I can dive deeper into the AWS REST API and I can exercise my Swift-fu in a challenging task.

The major challenge is performing a successful, authenticated S3 REST API request. The new Signature Version 4 signing process requires jumping through a few hoops to sign the request correctly.

The AWS documentation goes deep and explains all the steps, but to me, nothing beats seeing an actual, successful request in the terminal. Actually getting a 200 OK with curl took me two evenings of trial and error.

Here are my notes. Perhaps you will find them useful.

Why?

Why even bother with this? Who cares about the REST API? Wouldn’t a sane person just use the official SDKs and never get down to the REST level? Plus, there’s aws-cli if you want to do things in the terminal.

Perhaps. However, as was my case, an official SDK for your platform or programming language might not be available. Furthermore, the challenge to do everything by hand allowed me to understand AWS a tiny bit better and was a fun exercise in command line, curl, and HTTP.

What we need

For this exercise we will need:

I’m working on OSX 10.10 and I will be using the following:

The process

Our primary job is constructing the Authorization header. It will contain the signature for the request. Generating the signature is a process best understood in 5 distinct steps:

  1. Canonical request
  2. String to sign
  3. Generate user’s signing key
  4. Calculate signature
  5. Build the “Authorization” header

I will walk through the steps for the simplest possible S3 request, GET Bucket. It’s a good candidate because it carries no payload and only requires the basic headers to complete successfully.

Afterwards, I will leave a few notes on what is needed for more complex requests.

To simplify the steps, I will be constructing two text files, which contain the intermediate representations required for the signing process.

Overview

I created a bucket my-precious-bucket in “US Standard” region. I will be using the virtual-host based addressing scheme. The request I will be performing is thus:

Method
GET
Host
https://my-precious-bucket.s3.amazonaws.com
Headers
Authorization - will contain the request signature
Host - will contain my-precious-bucket.s3.amazonaws.com
x-amz-content-sha256 - required by AWS, must be the SHA256 digest of the payload (see below)
x-amz-date - required by AWS, must contain the timestamp of the request; the accepted format is quite flexible, I’m using ISO8601 basic format.

Example data used for all the calculations:

AWS access key ID
AKIAIOSFODNN7EXAMPLE
AWS secret access key
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Timestamp
20150915T124500Z (Tue, 15 Sep 2015 12:45:00 GMT)

The request will carry no payload (i.e. the body will be empty). This means that wherever a “payload hash” is required, we will provide an SHA256 hash of an empty string. And that is a constant value of e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855. This concerns the x-amz-content-sha256 header as well.

Step 1: Canonical request

First we need to prepare what’s called the “Canonical request”. This is a textual representation of the request we’re performing. The blueprint of this string is explained in the AWS docs and looks like this:

<HTTPMethod>\n
<CanonicalURI>\n
<CanonicalQueryString>\n
<CanonicalHeaders>\n
<SignedHeaders>\n
<HashedPayload>

It is our job to fill in the placeholders as required and describe the request we’re about to perform.

For GET Bucket, the result is as follows:

GET
/

host:my-precious-bucket.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20150915T124500Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Notes:

I type this into a plain text file as get-my-precious-bucket.creq. Note there is no newline character after the last line. We will be using this file in the following step.

Step 2: String to sign

The actual signature will be calculated for a particular string, called “string to sign”. The required format of this string looks like this:

AWS4-HMAC-SHA256\n
<Timestamp>\n
<Scope>\n
<CanonicalRequestHash>

Again, I will use a text editor to compose this string as a four-line text file. I will be calling it get-my-precious-bucket.sts:

AWS4-HMAC-SHA256
20150915T124500Z
20150915/us-east-1/s3/aws4_request
ef7c45cc2b0f100ea5d65024643f5cbaf83e7ba2717108905acd605cfe17bc6b

Notes:

The <CanonicalRequestHash> can be obtained with openssl:

$ openssl dgst -sha256 get-my-precious-bucket.creq
SHA256(get-my-precious-bucket.creq)= ef7c45cc2b0f100ea5d65024643f5cbaf83e7ba2717108905acd605cfe17bc6b

Again, remember - no newline at the end of the file.

Step 3: Generate user’s signing key

We have the string to sign, now we need the key which we’ll use to sign it. The signing key is derived from the user’s secret access key, through a four-step HMAC-SHA256 hashing process.

AWS explains the process with examples in a number of programming languages. I will just stick to bash and calculate the signing key using the openssl dgst command.

Here’s a simple script signing_key.sh to perform the calculation:

#!/bin/bash
function hmac_sha256 {
  key="$1"
  data="$2"
  echo -n "$data" | openssl dgst -sha256 -mac HMAC -macopt "$key" | sed 's/^.* //'
}

secret="$1"
date="$2"
region="$3"
service="$4"

# Four-step signing key calculation
dateKey=$(hmac_sha256 key:"AWS4$secret" $date)
dateRegionKey=$(hmac_sha256 hexkey:$dateKey $region)
dateRegionServiceKey=$(hmac_sha256 hexkey:$dateRegionKey $service)
signingKey=$(hmac_sha256 hexkey:$dateRegionServiceKey "aws4_request")

echo $signingKey

To run the script, provide your AWS secret access key, the date, region and service name. Example:

$ ./signing_key.sh wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY 20150915 us-east-1 s3
7b0b3063e375aa1e25890e0cae1c674785b8d8709cd2bf11ec670b96587650da

The output is a hexadecimal representation of the signing key. Hold on to it for the next step.

Step 4: Calculate signature

We have the string to sign and we have the signing key - now we need one final HMAC-SHA256 calculation to obtain the signature. I will use openssl dgst one last time:

$ openssl dgst -sha256 \
               -mac HMAC \
               -macopt hexkey:7b0b3063e375aa1e25890e0cae1c674785b8d8709cd2bf11ec670b96587650da \
               get-my-precious-bucket.sts
HMAC-SHA256(get-my-precious-bucket.sts)= 182072eb53d85c36b2d791a1fa46a12d23454ec1e921b02075c23aee40166d5a

The hash 182072eb... is the request signature. Hold on to it.

Step 5: Build the “Authorization” header

The calculated signature needs to be placed in the Authorization header and included in the HTTP request. The format of the header is as follows:

Authorization: <Algorithm> Credential=<Access Key ID/Scope>, SignedHeaders=<SignedHeaders>, Signature=<Signature>

We’ve already seen most of the values in previous steps. The only addition is the <Access Key ID>, where we place the AWS access key ID. In our example, the complete header looks like this:

Authorization: AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20150915/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=182072eb53d85c36b2d791a1fa46a12d23454ec1e921b02075c23aee40166d5a

Final step: Perform the request

We’re ready to curl the request. We need to include the Authorization header, as well as the two required x-amz-content-sha256 and x-amz-date headers. We also need the Host header, but curl will place this one for us automatically, based on the URL provided.

Here’s the complete call:

curl -v https://my-precious-bucket.s3.amazonaws.com/ \
     -H "Authorization: AWS4-HMAC-SHA256 \
         Credential=AKIAIOSFODNN7EXAMPLE/20150915/us-east-1/s3/aws4_request, \
         SignedHeaders=host;x-amz-content-sha256;x-amz-date, \
         Signature=182072eb53d85c36b2d791a1fa46a12d23454ec1e921b02075c23aee40166d5a" \
     -H "x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" \
     -H "x-amz-date: 20150915T124500Z"

If we’ve done everything right, we’ll receive the following (formatted for clarity):

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>my-precious-bucket</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>my-precious-file-1.txt</Key>
    <LastModified>2015-09-15T12:49:35.000Z</LastModified>
    <ETag>&quot;e243bb39c844b3543a7726576c869caf&quot;</ETag>
    <Size>7</Size>
    <Owner>
      <ID>948b6b7d545aaa9ed10fa52fcf06827960c607d47c96574101e3a68045fc872d</ID>
      <DisplayName>lukasz</DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
</ListBucketResult>

Moving further

If you’ve made it this far, congratulations! Correctly signing an AWS request is not trivial and there’s a reason why we use SDKs to do it for us.

The steps described above are the same for signing all authenticated S3 REST API requests. The simple example makes it easier to understand, but the process is the same throughout the API.

For more complex requests (e.g. PUT Object), keep the following in mind:

Troubleshooting

If you’re having trouble, first check the following:

Fortunately, error messages from the API are quite descriptive. For instance, if you fail to meet the 15 minutes time limit, you will receive a RequestTimeTooSkewed error.

If you have further problems, drop me an e-mail. I will do my best to help.