There seems to be a lot of general documentation on how to do OAuth but there wasn't much about using it for installed apps so what follows is an example using PHP that is basically what went into the plugin update.

It helps to first know the basic steps that will make this work. From the documentation those steps are:

1. Your application gets an unauthorized request token from Google's authorization server.
2. Google asks the user to grant you access to the required data.
3. Your application gets an authorized request token from the authorization server.
4. You exchange the authorized request token for an access token.
5. You use the access token to request data from Google's service access servers.

I will reference these steps in the code that follows.

Before trying this example out grab the OAuth PHP library. The only file that is needed to execute the examples is OAuth.php.

I have split the code into two files that I named phase1.php and phase2.php. In phase1.php the resulting URL would be used to grant access to the given scope (note that I've marked parts of the code that would need to be customized with a "Customize this" comment):

require_once("OAuth.php");

// This is the setup for step 1
// 1. Your application gets an unauthorized request token from Google's authorization server.
$signature_method = new OAuthSignatureMethod_HMAC_SHA1();

$params = array();

// The callback is where the user comes back after authentication
$params['oauth_callback'] = 'http://example.com/phase2.php'; // Customize this

// The scope is what you are asking for access to
$params['scope'] = 'https://www.google.com/analytics/feeds/'; // Customize this

// Setting xoauth_displayname will show the given name on the authentication screen
$params['xoauth_displayname'] = 'My Test App'; // Customize this

// Set up an anonymous oauth consumer
$consumer = new OAuthConsumer('anonymous', 'anonymous', NULL);

// Set up the request for a request token
$req_req = OAuthRequest::from_consumer_and_token($consumer, NULL, 'GET', 'https://www.google.com/accounts/OAuthGetRequestToken', $params);

// Sign the request
$req_req->sign_request($signature_method, $consumer, NULL);

// Set up curl and have it get the token to use for the authenication call
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $req_req->to_url());

// This tells curl to return the response as one string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 

// Run curl and grab the output and the return code
// This is the execution of step 1
$return = curl_exec($ch);
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);

if($http_code == 200)
{
  // If the call was good parse out the response parameters into an array
  $access_params = array();
  $param_pairs = explode('&', $return);
  foreach($param_pairs as $param_pair)
  {
    if (trim($param_pair) == '') { continue; }
    list($key, $value) = explode('=', $param_pair);
    $access_params[$key] = urldecode($value);
  }

  // Print out the authentication URL that needs to be opened in a browser
  echo "Open this link: https://www.google.com/accounts/OAuthAuthorizeToken?oauth_token=" . urlencode($access_params['oauth_token']) . "\n";

  // This is the oauth token secret that is needed for the second part
  echo "\nRemember the folowing value: " . $access_params['oauth_token_secret'] . "\n";

  // This is where the user would perform step 2
  // 2. Google asks the user to grant you access to the required data.
}
else
{
  echo "Error: $http_code and $return\n";
}

The above code performs steps 1 and 2. There are a few notes that are good to know:

• The OAuth consumer is set to "anonymous" here because this is an installed app, that is used again in the second phase.
• There can be multiple scope values, just seperate them with a space.
• Make sure to save the oauth_token_secret value, it is required for the second phase.

At this point the user goes off to authorize the application on Google's system. When they are done they are redirected back to your callback URL specified in phase 1 as oauth_callback. I put the following code in a file named phase2.php (again I've noted places that would need to be customized):

require_once("OAuth.php");

// This is where step 3 comes in, this would be in the callback and would be called
// after the user authenticates.
// 3. Your application gets an authorized request token from the authorization server.
$oauth_verifier = urldecode($_REQUEST['oauth_verifier']);
$oauth_token = urldecode($_REQUEST['oauth_token']);

// The token secret would need to be looked up from the previous step
$ouath_token_secret = ''; // Customize this

// This is where setup for step 4 starts
// 4. You exchange the authorized request token for an access token.
$signature_method = new OAuthSignatureMethod_HMAC_SHA1();

$params = array();

$params['oauth_verifier'] = $oauth_verifier;

// Set up the anonymous consumer again
$consumer = new OAuthConsumer('anonymous', 'anonymous', NULL);

// Use the oauth token and token secret to set up the consumer for the access token
$final_consumer = new OAuthConsumer($oauth_token, $ouath_token_secret);

// Set up the call to get the access token
$acc_req = OAuthRequest::from_consumer_and_token($consumer, $final_consumer, 'GET', 'https://www.google.com/accounts/OAuthGetAccessToken', $params);

$acc_req->sign_request($signature_method, $consumer, $final_consumer);

// Set up curl and have it get the final token and secret
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $acc_req->to_url());

// This tells curl to return the response as one string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 

// Run curl and grab the output and the return code
// This is the execution of step 4
$return = curl_exec($ch);
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);

if($http_code == 200)
{
  // If the call was good parse out the response parameters into an array
  $access_params = array();
  $param_pairs = explode('&', $return);
  foreach($param_pairs as $param_pair)
  {
    if (trim($param_pair) == '') { continue; }
    list($key, $value) = explode('=', $param_pair);
    $access_params[$key] = urldecode($value);
  }

  // Print out the final information needed for access to the service that was in "scope"
  echo "Access info: " . $access_params['oauth_token'] . " and " . $access_params['oauth_token_secret'] . "\n";
}
else
{
  echo "Error: $http_code and $return\n";
}

At this point you have a good oauth_token and oauth_token_secret that would need to be stored securely. These two bits of information allow you to make calls later to services provided under the given scope listed in the initial token request. Here is an example of making a request:

require_once("OAuth.php");

// This is step 5
// 5. You use the access token to request data from Google's service access servers.

$url = 'https://www.google.com/analytics/feeds/accounts/default';

$signature_method = new OAuthSignatureMethod_HMAC_SHA1();

$params = array();

$consumer = new OAuthConsumer('anonymous', 'anonymous', NULL);

// Here the saved oauth_token and oauth_token_secret are used
$token = new OAuthConsumer($saved_oauth_token, $saved_oauth_token_secret);

$oauth_req = OAuthRequest::from_consumer_and_token($consumer, $token, 'GET', $url, $params);

$oauth_req->sign_request($signature_method, $consumer, $token);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, $url);

// Use the oauth request to create the authenication header
curl_setopt($ch, CURLOPT_HTTPHEADER, array($oauth_req->to_header())));

curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

$return = curl_exec($ch);
$this->http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);

if($this->http_code == 200)
{
  echo $return;
}
else
{
  echo $return;
}

For a lot more detail check out Google's OAuth documentation. I also found it helpful to use the Google's OAuth playground to test things out.

I also found a handy list of all the define Google OAuth scopes.

First off you may ask yourself why you would want to use this. The best answer I have for that is that the solution in the other post loads the entire result set into memory from the very start while with this solution you can control just how many rows you load. To get started you will want to take a look at the MySQLi statement set attribute call. This call is will let you modify the underlying cursor type that is used with the prepared statement in two ways that are useful for this issue.

The MYSQLI_STMT_ATTR_CURSOR_TYPE attribute controls the type of cursor used for the results from the prepared statement. The default option is MYSQLI_CURSOR_TYPE_NO_CURSOR which results in no cursor being used at all. The only other option is MYSQLI_CURSOR_TYPE_READ_ONLY and this is the one you will want to use.

The MYSQLI_STMT_ATTR_PREFETCH_ROWS attribute controls the number of rows that are fetched under the covers from the server as more data is needed. The default for this attribute is 1 which means that each fetch of the row causes a round trip to the server. Larger numbers for this attribute will cause more memory to be used to store the resulting rows but fewer round trips to the server. This is the attribute that gives this option more flexibility.

The following is a slightly modified example that was used in the previous post now using both the MYSQLI_STMT_ATTR_CURSOR_TYPE and MYSQLI_STMT_ATTR_PREFETCH_ROWS attributes to efficiently retrieve rows from two different prepared statements at the same time:

<?php

  $db_connection = new mysqli('127.0.0.1', 'user', '', 'test');

  $post_stmt = $db_connection->prepare("select id, title from post where id between 10000 and 10999");
  $post_stmt->attr_set(MYSQLI_STMT_ATTR_CURSOR_TYPE, MYSQLI_CURSOR_TYPE_READ_ONLY);
  $post_stmt->attr_set(MYSQLI_STMT_ATTR_PREFETCH_ROWS, 100);
  $comment_stmt = $db_connection->prepare("select user_id from comment where post_id = ?");

  if ($post_stmt->execute())
  {
    $post_stmt->bind_result($post_id, $post_title);

    while ($post_stmt->fetch())
    {
      $comments = array();

      $comment_stmt->bind_param('i', $post_id);
      if ($comment_stmt->execute())
      {
        $comment_stmt->bind_result($user_id);
        while ($comment_stmt->fetch())
        {
          array_push($comments, array('user_id' => $user_id));
        }
      }
      else
      {
        printf("Comment statement error: %s\n", $comment_stmt->error);
      }

      printf("ID: %d -> %s\n", $post_id, $post_title);
      print_r($comments);
    }
  }
  else
  {
    printf("Post statement error: %s\n", $post_stmt->error);
  }

  $post_stmt->close();
  $comment_stmt->close();

  $db_connection->close();
?>

The first thing to note is that they are only supporting 64 bit systems officially. Having said that it isn't too hard to modify the code to make it work on a 32 bit system although it may turn out that such early modifications are missing some fundamental bits on why they were only support 64 bit systems. I'm going to assume at first that you are using a 64 bit system and then end with what you need if you are still using a 32 bit system.

I don't actually have a 64 bit system myself so I used an EC2 instance for the following instructions. To do the same start with Amazon's Basic 64-bit Fedora Core 8 (AMI Id: ami-86db39ef) instance (note that this is EBS backed so you will end up with an EBS volume after you start it) and then upgrade to Fedora 12 using my previous instructions on building a EBS bootable Fedora 12 instance. You will need to remove a few packages to get the 64 bit version of Fedora 8 to upgrade that I didn't have to do for the 32 bit version, here are all the commands you need to get to a running 64 bit Fedora 12 instance (the entire upgrade takes about 20 minutes):

# Fedora 8 to Fedora 10
yum -y remove dmraid-1.0.0.rc14-4.fc8.i386 dmraid-1.0.0.rc14-4.fc8.i386 curl-7.18.2-7.fc8.i386
yum clean all
rpm -Uhv http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-10-1.noarch.rpm http://archive.kernel.org/fedora-archive/releases/10/Fedora/i386/os/Packages/fedora-release-notes-10.0.0-1.noarch.rpm
yum -y update

# Fedora 10 to Fedora 11
yum -y remove gpm-1.20.5-2.fc10.i386
yum clean all
rpm -Uvh http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-11-1.noarch.rpm http://mirrors.usc.edu/pub/linux/distributions/fedora/linux/releases/11/Fedora/i386/os/Packages/fedora-release-notes-11.0.0-2.fc11.noarch.rpm
yum -y update

# Fedora 11 to Fedora 12
yum -y remove cryptsetup-luks-1.0.6-7.fc11.i586
yum clean all
rpm -Uvh http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-notes-12.0.0-4.fc12.noarch.rpm http://mirrors.kernel.org/fedora/releases/12/Fedora/i386/os/Packages/fedora-release-12-1.noarch.rpm
yum -y update

# Make sure the basics are installed
yum -y install gcc-c++ git

To start with there are some prerequisites you need. This can be taken care of in one command with yum:

yum -y install git cmake boost pcre-devel libicu-devel libmcrypt-devel oniguruma-devel mysql-devel gd-devel boost-devel libxml2-devel libcap-devel binutils-devel flex bison expat-devel

Next create a directory to hold everything in, change into that directory and create another directory to hold the customized libraries needed to compile HipHop PHP:

mkdir hiphop
cd hiphop
mkdir local

Next it is time to pull down the HipHop PHP source along with the source for some libraries it depends on (these all go into the hiphop directory created above):

git clone git://github.com/facebook/hiphop-php.git

wget "http://downloads.sourceforge.net/project/re2c/re2c/0.13.5/re2c-0.13.5.tar.gz?use_mirror=cdnetworks-us-2"
wget "http://www.threadingbuildingblocks.org/uploads/77/142/2.2/tbb22_20090809oss_src.tgz"
wget http://curl.haxx.se/download/curl-7.20.0.tar.bz2
wget http://www.monkey.org/~provos/libevent-1.4.13-stable.tar.gz

tar xvjf curl-7.20.0.tar.bz2
tar xvzf libevent-1.4.13-stable.tar.gz
tar xvzf re2c-0.13.5.tar.gz
tar xvzf tbb22_20090809oss_src.tgz

Next the customized patches get applied to some of the library sources and each is built to install in the custom directory:

export CMAKE_PREFIX_PATH=`pwd`/local

cd tbb22_20090809oss
gmake
cp -Rp include/tbb/ /usr/include/
cp `pwd`/build/*_release/*.so /usr/lib/
ldconfig
cd ..

cd re2c-0.13.5
./configure --prefix=`pwd`/../local
make install
cd ..

cd libevent-1.4.13-stable
cp ../hiphop-php/src/third_party/libevent.fb-changes.diff .
patch < libevent.fb-changes.diff
./configure --prefix=`pwd`/../local
make install
cd ..

cd curl-7.20.0
cp ../hiphop-php/src/third_party/libcurl.fb-changes.diff .
patch -p0 < libcurl.fb-changes.diff
./configure --prefix=`pwd`/../local
make install
cd ..

There is one problem at this point that requires a little surgery on the HipHop PHP source itself. There is more about this in issue #6 and once it gets fixed this won't need to be done.

cd hiphop-php
echo "#ifndef LHASH" >> src/cpp/ext/ext_openssl.h
echo "#define LHASH LHASH_OF(CONF_VALUE)" >> src/cpp/ext/ext_openssl.h
echo "#endif" >> src/cpp/ext/ext_openssl.h

And at last it is time to compile HipHop PHP itself:

git submodule init
git submodule update
export HPHP_HOME=`pwd`
export HPHP_LIB=`pwd`/bin
cmake .
make

It takes about 20 minutes to compile everything. Once the compile is done you are ready to roll. Check out the running HipHop wiki page to learn how to run the resulting binary. One important thing to note is that you need to make sure you have the correct environment variables set when you go to compile things. I created a little file I can source with the following in it:

export HPHP_BASE=<path to the first directory>
export CMAKE_PREFIX_PATH=$HPHP_BASE/local
export HPHP_HOME=$HPHP_BASE/hiphop-php
export HPHP_LIB=$HPHP_HOME/bin

For those who just want it to go I've put all of the above into one script that can be found here. If you are going from Fedora 8 to Fedora 12 on an EC2 node you can get a script for that here.

Now if you want to do this on a 32 bit Fedora 12 install you will need to modify the source first. The easiest way I know of doing this is to look at this commit log or clone my version that can be found here:

git clone git://github.com/carsonmcdonald/hiphop-php.git

Please note that my version my not be up to date and the modifications to get the source to build on the 32 bit system may not be 100% correct. My goal was to get it to build and run on a 32 bit system but I don't have the time to very much more than that.

"Commands out of sync; you can't run this command now"

Details about this error can be found in the mysql docs. Reading those details makes it clear that the result sets of a prepared statement execution need to be fetched completely before executing another prepared statement on the same connection.

Fixing the issue can be accomplished by using the store result call. Here is an example of what I initially was trying to do:

<?php

  $db_connection = new mysqli('127.0.0.1', 'user', '', 'test');

  $post_stmt = $db_connection->prepare("select id, title from post where id = 1000");
  $comment_stmt = $db_connection->prepare("select user_id from comment where post_id = ?");

  if ($post_stmt->execute())
  {
    $post_stmt->bind_result($post_id, $post_title);

    if ($post_stmt->fetch())
    {
      $comments = array();

      $comment_stmt->bind_param('i', $post_id);
      if ($comment_stmt->execute())
      {
        $comment_stmt->bind_result($user_id);
        while ($comment_stmt->fetch())
        {
          array_push($comments, array('user_id' => $user_id));
        }
      }
      else
      {
        printf("Comment statement error: %s\n", $comment_stmt->error);
      }
    }
  }
  else
  {
    printf("Post statement error: %s\n", $post_stmt->error);
  }

  $post_stmt->close();
  $comment_stmt->close();

  $db_connection->close();

  printf("ID: %d -> %s\n", $post_id, $post_title);
  print_r($comments);
?>

The above will result in the following error:

Here is what needs to be done to make it work correctly:

<?php

  $db_connection = new mysqli('127.0.0.1', 'user', '', 'test');

  $post_stmt = $db_connection->prepare("select id, title from post where id = 1000");
  $comment_stmt = $db_connection->prepare("select user_id from comment where post_id = ?");

  if ($post_stmt->execute())
  {
    $post_stmt->store_result();
    $post_stmt->bind_result($post_id, $post_title);

    if ($post_stmt->fetch())
    {
      $comments = array();

      $comment_stmt->bind_param('i', $post_id);
      if ($comment_stmt->execute())
      {
        $comment_stmt->bind_result($user_id);
        while ($comment_stmt->fetch())
        {
          array_push($comments, array('user_id' => $user_id));
        }
      }
      else
      {
        printf("Comment statement error: %s\n", $comment_stmt->error);
      }
    }

    $post_stmt->free_result();
  }
  else
  {
    printf("Post statement error: %s\n", $post_stmt->error);
  }

  $post_stmt->close();
  $comment_stmt->close();

  $db_connection->close();

  printf("ID: %d -> %s\n", $post_id, $post_title);
  print_r($comments);
?>

A couple things to note about the above example:

• The bind and fetch on the statement still works correctly.
• Make sure the results are freed when the processing is done.

This post goes over how I went about configuring Sphinx and gives an example of how to integrate it with PHP. I'm using MySQL as the data store filled with the Stack Overflow CC data dump although it should be easy to adapt the instructions to other data sources. To follow along just download a copy of the data dump and use my schema and loader to get the same MySQL database.

I've broken the setup down into the following 4 steps:

1. Configuring Sphinx
2. Building an Index with Sphinx
3. Searching with Sphinx
4. Using the Sphinx PHP API

As a side note, Sphinx is available as a package under Fedora 11 and I assume the same of other Linux distributions so it shouldn't be too hard to install. I am using version 0.9.8.1 compiled from source.

Configuring Sphinx

The Sphinx configuration file can be be broken down into different parts with some of those parts being used by the indexer and some by the search service. I have broken the configuration file I am using into different sections but I include the full file for download at the end. The first segment of the configuration describes where to get the data for the index, in this case it is a MySQL database:

There are a few notable parts to the above. The field "sql_query" defines the query that grabs data from the database for the index. The "sql_attr_*" values point out the columns in the data that will be used in search queries later to sort or group by, more on that in the searching section.

The next section defines the index:

In the above there are two index definitions. The first index is very basic and has no morphology defined. The second index is derived from the first index and uses a stemming morphology. I'm also having any HTML stripped out of the content since there is some in the data and it doesn't need to be searchable in the index.

The next block in the configuration file defines how the indexer will run. Here I'm limiting the amount of memory used to 32M as an example of what could be done:

The final part of the configuration file sets options for the search daemon:

You can download the full configuration file here: sphinx.conf

The above configuration gets you the basics. If you want to find out more you can check out the indexing documentation.

Building an Index with Sphinx

The complexity of building and maintaining an index goes up with the frequency of the updates that are required to keep it up to date. Luckily the data set I'm working with gets updated once a month so it is fairly static. Because the data is updated monthly I only have to update the index once a month so this makes the indexing simple.

This is the command I use to create the index:

indexer --all --config sphinx.conf

Indexing the entire data set I have only takes a few minutes so re-indexing speed isn't an issue for me and the resulting index is in the 250M range as of this post so size isn't necessarily an issue either. However I currently create the index on one machine and then upload it to the server so a 250M transfer could become a bandwidth hog if I needed to update the index more often. If I were able to get deltas from the data dump I would look into using the index merging feature and then transferring the delta index.

Searching with Sphinx

Sphinx offers a lot of flexibility in configuring how a search is run and I found myself needing that flexibility to get better results. The search command line interface (see the reference for all the options) can be used to test a few of the options but you can't do everything with it that you can with the various APIs available.

The first place I started changing defaults with the search is in how it matches the search words. The default is to find all words:

search --config sphinx.conf "php full text search"

Here is an example of the output you get from the CLI search:

The output shows you the "document id" as well as the weight and any "sql_attr_" values. I used the CLI search to get the weights for each search change I made before I started working with the API. Please note that the extended information in the above example output only shows up if the sql_query_info query is set correctly in the configuration file.

Next I tested with "any" and "extended search version 2" before settling on extended version 2:

search --any --config sphinx.conf "php full text search"

search --ext2 --config sphinx.conf "php full text search"

The CLI search will also perform sorts. It is important to note that you can only sort on values that were indexed using the "sql_attr_"* configuration options. Here is an example of sorting by an indexed value:

search --sortby "date_added desc" --config sphinx.conf "php regex"

This is what happens when you try to sort based on an un-index attribute:

search --sortby "id desc" --config sphinx.conf "php regex"

Sphinx 0.9.8.1-release (r1533)
Copyright (c) 2001-2008, Andrew Aksyonoff

using config file 'sphinx.conf'...
index 'so_2009_10': search error: failed to create sorting queue: sort-by attribute 'id' not found.

One of the things that the CLI search interface can't do is change weightings for the different data points. I found that weighting the title more than the body made sense and that is reflected in the PHP code that follows in the API section.

The last part of searching to cover is searchd server. This is the integration point for the various Sphinx APIs and I assume it is running for the examples in the PHP API section. If the configuration example given in the first section is used all that needs to be done is to start the daemon. The following is a simple init script I use to start and stop searchd:

#!/bin/sh

case "$1" in
'start')
searchd --config /etc/sphinx/sphinx.conf
;;
'stop')
killall -9 searchd;
;;
*)
echo "Usage: $0 { start | stop}"
exit 1
;;
esac
exit 0
#

Using the Sphinx PHP API

The Sphinx PHP API is included in the Sphinx source. The API is contained in one file named sphinxapi.php that is located in the api directory of the source. Make sure the library is in a place where it can be included before trying out the examples. There is also some documentation for the PHP API.

In the following example I'm running the search query then serializing it to JSON format. Notice that the title is given a weight of 70 and the body_text is given a weight of 30 so the results will focus more on the title than what is in the body:

Although I picked PHP there are a wide range of language specific libraries available.

Overall it didn't seem too difficult to set up Sphinx and have it serving search results quickly. I liked that it is very light weight and doesn't need a lot of resources or require a lot of extra parts to be installed.

With the patch you can now do stuff like:

Before you had to use a funky function generation call.

Tags: php

For the longest time I've been using the older PHP JSON library to do JSON with PHP but now that the extension is included in the core I decided it was time to test it out. I took a couple of minutes and converted my DNSBL checker as a test since it has a fairly large data-set that gets converted and sent back. The json functions provided by the extension are probably easier to use since the JSON library needed you to create an object first but that wasn't a real issue. After making the change I could tell JSON extension was faster than the library. After a little digging I found that someone has done a little extension vs library testing and claims the JSON extension is 153 times as fast as the library.

Tags: php, ajax

To clear up the expiration time issue I think these two steps are needed:

1) Keep in mind that the HTTP header dates must be in GMT.
2) The PHP function time() returns the seconds since the epoch January 1 1970 00:00:00 GMT). Notice here this is in GMT as well.
3) The HTTP Date header you see in a response from an S3 server is the time on that server. The machine you use to sign your request should be synced with that time. I think a good guess is that all the Amazon servers are synced with the atomic clock.

There isn't much to securing a URL after you have that tucked away. Here is an example that will sign a URL so that it is valid for 60 seconds:

The hex2b64 function was pulled from the amazon S3 PHP example library.

Tags: amazon s3, s3, php