Lately I’ve been more and more attracted to the AWS platform. My current pet project is a simple OS X screenshot sharing app. S3 instantly felt like the appropriate storage for the files.
At the moment, there is no official AWS SDK for Mac. In a way this makes my task more interesting. I can dive deeper into the AWS REST API and I can exercise my Swift-fu in a challenging task.
The major challenge is performing a successful, authenticated S3 REST API request. The new Signature Version 4 signing process requires jumping through a few hoops to sign the request correctly.
The AWS documentation goes deep and explains all the steps, but to me, nothing
beats seeing an actual, successful request in the terminal. Actually getting a
200 OK with
curl took me two evenings of trial and error.
Here are my notes. Perhaps you will find them useful.
Why even bother with this? Who cares about the REST API? Wouldn’t a sane person just use the official SDKs and never get down to the REST level? Plus, there’s aws-cli if you want to do things in the terminal.
Perhaps. However, as was my case, an official SDK for your platform or programming language might not be available. Furthermore, the challenge to do everything by hand allowed me to understand AWS a tiny bit better and was a fun exercise in command line, curl, and HTTP.
What we need
For this exercise we will need:
- a way to calculate a SHA256 digest of a string
- a way to calculate a HMAC-SHA256 mac of a string (with both a string key and a binary/hexadecimal key)
- a text editor (make sure you can omit trailing newline in a file)
I’m working on OSX 10.10 and I will be using the following:
openssl1.0.2 (you need at least
openssl1.0.0 for the
Our primary job is constructing the
Authorization header. It will contain the
signature for the request. Generating the signature is a step best understood in
5 distinct steps:
- Canonical request
- String to sign
- Generate user’s signing key
- Calculate signature
- Build the “Authorization” header
I will walk through the steps for the simplest possible S3 request, GET Bucket. It’s a good candidate because it carries no payload and only requires the basic headers to complete successfully.
Afterwards, I will leave a few notes on what is needed for more complex requests.
To simplify the steps, I will be constructing two text files, which contain the intermediate representations required for the signing process.
I created a bucket
my-precious-bucket in “US Standard” region. I will be using
the virtual-host based addressing scheme.
The request I will be performing is thus:
Authorization- will contain the request signature
Host- will contain
x-amz-content-sha256- required by AWS, must be the SHA256 digest of the payload (see below)
x-amz-date- required by AWS, must contain the timestamp of the request; the accepted format is quite flexible, I’m using ISO8601 basic format.
Example data used for all the calculations:
- AWS access key ID
- AWS secret access key
20150915T124500Z(Tue, 15 Sep 2015 12:45:00 GMT)
The request will carry no payload (i.e. the body will be empty). This means that
wherever a “payload hash” is required, we will provide an SHA256 hash of an
empty string. And that is a constant value of
x-amz-content-sha256 header as well.
First we need to prepare what’s called the “Canonical request”. This is a textual representation of the request we’re performing. The blueprint of this string is explained in the AWS docs and looks like this:
It is our job to fill in the placeholders as required and describe the request we’re about to perform.
GET Bucket, the result is as follows:
<CanonicalQueryString>is an empty string. Note however that the line must not be omitted and the newline is required.
<CanonicalHeaders>contain the HTTP headers we will be putting in the request, one per line. The names of the headers are lowercased and whitespace is stripped. In this very basic case, the only required headers are
<SignedHeaders>is a semicolon-separated list of the headers. We just repeat the ones we used for
<HashedPayload>is the SHA256 checksum of the request payload. Since we carry no payload in this request, we append the SHA256 hash of an empty string.
I type this into a plain text file as
get-my-precious-bucket.creq. Note there
is no newline character after the last line. We will be using this file in the
The actual signature will be calculated for a particular string, called “string to sign”. The required format of this string looks like this:
Again, I will use a text editor to compose this string as a four-line text file.
I will be calling it
AWS4-HMAC-SHA256is a constant string which needs to be placed in the first line. It specifies the hash algorithm used for request signing.
<Timestamp>is the current UTC time in ISO 8601 basic format.
<Scope>binds the request to a certain date, AWS region, and service. It must be in the format:
aws4_requestis a constant string.
<CanonicalRequestHash>is a hexadecimal representation of the SHA256 hash of our canonical request - the string from Step 1.
<CanonicalRequestHash> can be obtained with
Again, remember - no newline at the end of the file.
We have the string to sign, now we need the key which we’ll use to sign it. The signing key is derived from the user’s secret access key, through a four-step HMAC-SHA256 hashing process.
explains the process
with examples in a number of programming languages. I will just stick to bash
and calculate the signing key using the
openssl dgst command.
Here’s a simple script
signing_key.sh to perform the calculation:
To run the script, provide your AWS secret access key, the date, region and service name. Example:
The output is a hexadecimal representation of the signing key. Hold on to it for the next step.
We have the string to sign and we have the signing key - now we need one final
HMAC-SHA256 calculation to obtain the signature. I will use
openssl dgst one
182072eb... is the request signature. Hold on to it.
The calculated signature needs to be placed in the
Authorization header and
included in the HTTP request. The format of the header is as follows:
We’ve already seen most of the values in previous steps. The only addition is
<Access Key ID>, where we place the AWS access key ID. In our
example, the complete header looks like this:
Final step: Perform the request
We’re ready to
curl the request. We need to include the
header, as well as the two required
headers. We also need the
Host header, but
curl will place this one for us
automatically, based on the URL provided.
Here’s the complete call:
If we’ve done everything right, we’ll receive the following (formatted for clarity):
If you’ve made it this far, congratulations! Correctly signing an AWS request is not trivial and there’s a reason why we use SDKs to do it for us.
The steps described above are the same for signing all authenticated S3 REST API requests. The simple example makes it easier to understand, but the process is the same throughout the API.
For more complex requests (e.g. PUT Object), keep the following in mind:
- Your request might have a nonempty body (e.g. the file to upload), so the
<HashedPayload>line will be based on that. You can use
openssl dgst -sha256 picture.jpgto get the digest of
- If you include other headers in the request (e.g.
Content-Typefor PUT Object), you also need to include them (in lowercase!) in the Canonical Request representation.
If you’re having trouble, first check the following:
- Ensure you’re not including whitespace where it doesn’t belong (or the other way around)
- Use the appropriate key in the appropriate place
- AWS access key ID goes into the
Credentialpart of the
- AWS secret access key is used to generate your signing key
- AWS access key ID goes into the
- Your request must be performed within 15 minutes of the specified timestamp. Otherwise it will be rejected by the server.
Fortunately, error messages from the API are quite descriptive. For
instance, if you fail to meet the 15 minutes time limit, you will receive a
If you have further problems, let me know in the comments. I will do my best to help.