Direct Browser Uploading – Amazon S3, CORS, FileAPI, XHR2 and Signed PUTs

I've been hacking around with FileAPI and XHR2 in HTML5 recently (more on why hopefully in another month or so). So when Amazon announced S3 CORS support I figured I should create a demo of directly uploading a file to S3 from a browser.

The first thing to understand is that while the upload happens directly to S3 there still needs to be some server side code that signs the URL used by the PUT call. That bit of code is really simple and I'm including an example at the end for both PHP and Ruby. If you want to skip to the fun part you can check out the PHP and Ruby example code on github (instructions there on deploying to Heroku as well).

Second there are a good number of technologies involved here so I've compiled a list of helpful links in case you aren't already familiar with them and/or want a reference:

Setting up CORS support for an S3 bucket can be done using the console, see the S3 CORS support docs above for details. For everything in this demo I used the following CORS configuration:

<CORSConfiguration>
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>PUT</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>Content-Type</AllowedHeader>
        <AllowedHeader>x-amz-acl</AllowedHeader>
        <AllowedHeader>origin</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

That configuration allows any origin to issue PUTs and include the headers Content-Type, x-amz-acl and origin. You would probably want to restrict the origin more but for this demo I want to make sure it works for people who just cut and paste the above.

The following HTML sets up the file input tag and a progress bar to track the upload (index.html):

<html>
<head>
 <link rel="stylesheet" type="text/css" href="styles.css" />
 <script type="text/javascript" src="app.js"></script>
</head>

<body>
  <table>
    <tr>
      <td>File selection:</td>
      <td><input type="file" id="files" name="files[]" multiple /></td>
    </tr>
    <tr>
      <td>Progress:</td>
      <td><div id="progress_bar"><div class="percent">0%</div></div></td>
    </tr>
    <tr>
      <td>Status:</td>
      <td><span id="status"></span></td>
    </tr>
  </table>

  <script type="text/javascript">
    document.getElementById('files').addEventListener('change', handleFileSelect, false);
    setProgress(0, 'Waiting for upload.');
  </script>

</body>

</html>

CSS for the progress bar (styles.css):

#progress_bar {
  width: 200px;
  margin: 10px 0;
  padding: 3px;
  border: 1px solid #000;
  font-size: 14px;
  clear: both;
  opacity: 0;
  -moz-transition: opacity 1s linear;
  -o-transition: opacity 1s linear;
  -webkit-transition: opacity 1s linear;
}
#progress_bar.loading {
  opacity: 1.0;
}
#progress_bar .percent {
  background-color: #99ccff;
  height: auto;
  width: 0;
}

The JavaScript that follows has a couple different parts.

  • handleFileSelect – This is where things get started when a file is selected for upload. It kicks off the upload process with each file that was selected.
  • uploadFile – Called for each file in handleFileSelect and ties the signing process to the S3 PUT process.
  • executeOnSignedUrl – Calls the server side signing process with the a filename and mime type. The server side signed URL is then passed on to a callback.
  • uploadToS3 – Uses a signed PUT URL to upload the given file to S3 using CORS enabled XHR2.
  • createCORSRequest – Creates a CORS XHR2 request.
  • setProgress – Sets the current progress of the upload.
function createCORSRequest(method, url) 
{
  var xhr = new XMLHttpRequest();
  if ("withCredentials" in xhr) 
  {
    xhr.open(method, url, true);
  } 
  else if (typeof XDomainRequest != "undefined") 
  {
    xhr = new XDomainRequest();
    xhr.open(method, url);
  } 
  else 
  {
    xhr = null;
  }
  return xhr;
}

function handleFileSelect(evt) 
{
  setProgress(0, 'Upload started.');

  var files = evt.target.files; 

  var output = [];
  for (var i = 0, f; f = files[i]; i++) 
  {
    uploadFile(f);
  }
}

/**
 * Execute the given callback with the signed response.
 */
function executeOnSignedUrl(file, callback)
{
  var xhr = new XMLHttpRequest();
  xhr.open('GET', 'signput.php?name=' + file.name + '&type=' + file.type, true);

  // Hack to pass bytes through unprocessed.
  xhr.overrideMimeType('text/plain; charset=x-user-defined');

  xhr.onreadystatechange = function(e) 
  {
    if (this.readyState == 4 && this.status == 200) 
    {
      callback(decodeURIComponent(this.responseText));
    }
    else if(this.readyState == 4 && this.status != 200)
    {
      setProgress(0, 'Could not contact signing script. Status = ' + this.status);
    }
  };

  xhr.send();
}

function uploadFile(file)
{
  executeOnSignedUrl(file, function(signedURL) 
  {
    uploadToS3(file, signedURL);
  });
}

/**
 * Use a CORS call to upload the given file to S3. Assumes the url
 * parameter has been signed and is accessable for upload.
 */
function uploadToS3(file, url)
{
  var xhr = createCORSRequest('PUT', url);
  if (!xhr) 
  {
    setProgress(0, 'CORS not supported');
  }
  else
  {
    xhr.onload = function() 
    {
      if(xhr.status == 200)
      {
        setProgress(100, 'Upload completed.');
      }
      else
      {
        setProgress(0, 'Upload error: ' + xhr.status);
      }
    };

    xhr.onerror = function() 
    {
      setProgress(0, 'XHR error.');
    };

    xhr.upload.onprogress = function(e) 
    {
      if (e.lengthComputable) 
      {
        var percentLoaded = Math.round((e.loaded / e.total) * 100);
        setProgress(percentLoaded, percentLoaded == 100 ? 'Finalizing.' : 'Uploading.');
      }
    };

    xhr.setRequestHeader('Content-Type', file.type);
    xhr.setRequestHeader('x-amz-acl', 'public-read');

    xhr.send(file);
  }
}

function setProgress(percent, statusLabel)
{
  var progress = document.querySelector('.percent');
  progress.style.width = percent + '%';
  progress.textContent = percent + '%';
  document.getElementById('progress_bar').className = 'loading';

  document.getElementById('status').innerText = statusLabel;
}

The above example calls the PHP version of the server side signing code. It can easily be changed to anything that can sign a request in the same way.

I have an old way of creating signed URLs using PHP that hasn't been updated in forever. With the more recent versions of PHP there is built in support for the hash-hmac function and a base64 encode. Here is the updated PHP script you need on the server side, to get it to work you would need to replace the S3_KEY, S3_SECRET and S3_BUCKET values with your own:

//
// Change the following settings
//
$S3_KEY='S3 Key Here';
$S3_SECRET='S3 Secret Here';
$S3_BUCKET='/uploadtestbucket';

$EXPIRE_TIME=(60 * 5); // 5 minutes
$S3_URL='http://s3.amazonaws.com';

$objectName='/' . $_GET['name'];

$mimeType=$_GET['type'];
$expires = time() + $EXPIRE_TIME;
$amzHeaders= "x-amz-acl:public-read";
$stringToSign = "PUT\n\n$mimeType\n$expires\n$amzHeaders\n$S3_BUCKET$objectName";
$sig = urlencode(base64_encode(hash_hmac('sha1', $stringToSign, $S3_SECRET, true)));

$url = urlencode("$S3_URL$S3_BUCKET$objectName?AWSAccessKeyId=$S3_KEY&Expires=$expires&Signature=$sig");

echo $url;

With all of that in place you should now be able to upload directly to S3 using a browser that supports CORS, XHR2 and the FileAPI (pretty much everything but IE currently).

This entry was posted in programming

7 Comments

  1. ChrisHF

    You say "there still needs to be some server side code that signs the URL used by the PUT call". I don't understand why that is. Is it to avoid putting your credentials in the JS? What if the page is for personal use only? What if the credentials are entered by the user? Then can you sign in the client? Or is there still some technical limitation?

  2. @ChrisHF You wouldn't want to keep your S3 secret in Javascript or anyone could grab it and have full control of your S3 account. You could certainly do it for personal use and I did think about a version that would let the user enter their own S3 information but I didn't know how useful that would be. So there shouldn't be any technical limitation but only a security limitation.

  3. Thomas

    Unfortunately doesn't seem to work in Firefox & IE .. greetz

  4. Yeah IE doesn't support CORS (details), XHR2 (details) or the FileAPI (details) yet. I'll have to look into the issues with Firefox because the latest version should be working.

  5. Thiago

    Hi. Thanks for the code. But Im trying to make a CORS PUT using your very same code and always getting a "403 forbiden" in the options request by the browser. Do you have any hints? ( my CORS rule on the bucket is exactly the same as yours )

  6. @Thiago @Thomas it turns out that S3 is currently authenticating the OPTION call that is made in "preflight" for CORS and that fails (probably because the auth has been signed with for a PUT request not an OPTION request) see https://forums.aws.amazon.com/thread.jspa?messageID=378235 for more.

    It looks like there was some debate by the browser devs on if this is actually correct or not, see https://bugzilla.mozilla.org/show_bug.cgi?id=778548 and https://bugs.webkit.org/show_bug.cgi?id=92755 There was no consensus on ignoring the non-200 response status even though the spec says not to ignore it. However it does seem like there was consensus that the server side should not attempt to authenticate an OPTION request since it is not made to be part of the auth. The main issue they hung on was legacy IIS web server not being able to authenticate only certain requests.

    Hopefully because the request already returns the correct information outside of the status code it will be fixed by just returning a 200 regardless of the authentication working or not.

  7. Thiago

    It seems that its really a amazon bug ( a aws developer posted there ). Thanks !!!

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>