Wednesday, May 18, 2011

Google REST Web Service Security C# URL Authentication

Google URL Signing
 
To access certain Google API web services a Client ID and a digital signature is required.  A Client ID and a cryptographic key (used to generate a unique digital signature) are provided by Google once consumers have created an account.  Before a REST web service request can be made, a digital signature must be generated and passed in as part of the URL, this process is known as URL signing.  The cryptographic key, sometimes called shared secret/signing key must be kept secret and is never passed in as part of a URL request.

The following are required before a digital signature can be generated:

  • A signing string (made up of the path and query of the request URL).
  • A signing key (cryptographic key provided by Google).
  • The correct encryption algorithm (Hash Message Access Control eg. HMAC SHA1, HMAC SHA2).

To generate a digital signature requires the following 3 steps.

1)  Construct request URL, the URL must be percent-encoded/URL encoded to ensure it is valid.

Constructing a valid URL

A URL is made up of reserved and unreserved characters.  Reserved characters in a URL have a specific purpose, for example a ‘/ ’ separates the different parts of a URL.  If a ‘/ ’ is required in a URL for any other purpose, then it must be percent-encoded to distinguish it from its reserved counterparts.  Percent-encoded reserved characters are always preceded by a ‘%’ to indicate percent-encoding.

The following are reserved characters:

!  * ' ( ) ; : @ & = + $ , / ? % # [ ]

To percent-encode a URL’s reserved characters, use the pair of hex digits which corresponds to each reserved character’s ASCII value, finish by preceding each hex digit pair with a ‘%’.  If any reserved character is non-ASCII then use the pair of hex digits which corresponds to its UTF-8 character instead.

For example, to percent-encode a '/ ':

a)  convert to or look up the ASCII value for a ‘/ ’, the ASCII value is 47

b)  next look up the pair of hex digits which correspond to this ASCII value, the hex digit pair is 2F

c)  next precede with a ‘%’, the result is %2F

URL encoding the following URL:

http://maps.googleapis.com/maps/api/geocode/json ?address=East 25th St & 3rd Ave&sensor=false&client=yourClientID

The address parameter contains spaces, spaces are not allowed in a URL, it also contains a '&'.  This URL must be encoded by replacing spaces with a '+', and replacing '&' with the two hex digits corresponding to its ASCII value.

Encoding would produce the following URL:

http://maps.googleapis.com/maps/api/geocode/json?address=East+25th+St+%26+3rd+Ave&sensor=false&client=yourClientID

For more information on ASCII characters and their corresponding pair of hex digits see the following link:


To see a list of reserved/unreserved characters and additional information on URL usage, go to the following link.


2)  Using the encoded URL

http://maps.googleapis.com/maps/api/geocode/json?address=East+25th+St+%26+3rd+Ave&sensor=false&client=yourClientID

Omit the protocol and host, leaving only the path and query, now you have the signing string.

/maps/api/geocode/json?address=East+25th+St+%26+3rd+Ave&sensor=false&client=yourClientID

3)  Generate signature using signing key, signing string and encryption algorithm.

Google provides a C# method for developers called ‘Sign’, this method accepts two string parameters and returns the signed URL request in full as a string.  When calling this method the full URL and the signing key are passed in as the two parameters of type string, see the method below.

Google’s C# 'Sign' method code snippet

public static string Sign(string url, string keyString)
{
      ASCIIEncoding encoding = new ASCIIEncoding();
      // converting key to bytes will throw an exception, need to
      // replace '-' and  '_' characters first.

      string usablePrivateKey = keyString.Replace("-", "+").Replace("_", "/");

      byte[] privateKeyBytes = Convert.FromBase64String(usablePrivateKey);

      Uri uri = new Uri(url);

      byte[] encodedPathAndQueryBytes = encoding.GetBytes(uri.LocalPath + uri.Query);

      // compute the hash

      HMACSHA1 algorithm = new HMACSHA1(privateKeyBytes);

      byte[] hash = algorithm.ComputeHash(encodedPathAndQueryBytes);

      // convert the bytes to string and make url-safe by replacing '+' and '/' characters

      string signature = Convert.ToBase64String(hash).Replace("+", "-").Replace("/", "_");

      // Add the signature to the existing URI.

      return uri.Scheme+"://"+uri.Host+uri.LocalPath + uri.Query +"&signature=" + signature;
}

A signature parameter is attached to the end of the URL, and the fully signed request URL is returned as a string.

http://maps.googleapis.com/maps/api/geocode/json?address=East+25th+St+%26+3rd+Ave&sensor=false&client=yourClientID&signature=HMACgeneratedSignature

Using this process a request can be made to any Google web service which requires a digital signature.

To download Google's C# console application for URL signing, which includes the 'Sign' method shown above, see the following link:


In some cases SSL (Secure Socket Layer) is used to provide transport level security when making requests. In these situations change the protocol from ‘http’ to ‘https’, before making the request.

Google Maps API Web Services require URL Authentication, documentation is provided on the URL signing process, how to produce valid URL's, SSL Access and processing XML results.  See the following link for more information:

Yahoo REST Web Service Security

Yahoo REST Web Service Authentication

All Yahoo API web services are free to consume at present.  Once service consumers have signed up for an account with Yahoo, they are assigned an API Key/Application ID.  When making a REST web service call the API Key must be included in the URL request.

Some Yahoo API web services use basic http authentication if a username and password is required, the Delicious API from Yahoo is an example.  The http Authorization header is assigned the value of the base64 encoded username and password.  A client with the following credentials:

username = MyUserName
password = MyPsswd

Places the username and password together to create the following string, which is then encoded using base64.

MyUserName:MyPsswd

The resulting base64 encoded string is passed as a http header during the request, for example:

Authorization: Basic TXlVc2VyTmFtZTpNeVBzc3dk

The Delicious API also uses Secure Socket Layer (SSL) for transport level security, the https protocol is used instead of http when making a request.  Some service providers use HTTP-Auth and SSL in combination.

Amazon Web Services (AWS) URL Authentication

There are two types of web service requests that can be made to Amazon Web Services (AWS).

  • Anonymous requests are made to freely available services using an Access Key ID which is issued when consumers sign up for an account.  This key is included as a request parameter in all requests made to AWS.
  • Authenticated requests must contain a request signature, in other words the URL must be signed using a generated digital signature.

    AWS Access Identifiers are an Access Key ID which is your identity/username and a Secret Access Key which is your password.  The Secret Access Key must be kept secret and is never used in requests, it is instead used together with an encryption algorithm during the signing process.  The Secret Access Key is a 40 character alphanumeric sequence issued to consumers by AWS.



    A Recipe for Signing URL Request’s to AWS

    1)  Construct your request URL.


    The following is a dummy URL

    http://aws.amazon.com/onca/xml?Service=AWSDummyService&AWSAccessKeyId=00000000&Operation=DummySearch&ResponseGroup=DummyInfo,TopSellers,NewReleases&Version=2011-05-05

    2)  TimeStamp your URL.

    Attach a time stamp to the end of your URL.

    http://aws.amazon.com/onca/xml?Service=AWSDummyService&AWSAccessKeyId=00000000&Operation=DummySearch&ResponseGroup=DummyInfo,TopSellers,NewReleases&Version=2011-05-05&TimeStamp=2011-5-03T14:22:58Z

    For more info on UTC format TimeStamp, see the link below.

    http://en.wikipedia.org/wiki/Unix_time

    3)  URL encode any reserved characters used by query parameters.

    The ResponseGroup parameter contains comma’s and the TimeStamp parameter contains colons, comma’s and colons are reserved characters. To URL encode a reserved character, use the hex digit pair corresponding to the reserved character’s ASCII value, then precede the hex digit pair with a ‘%’.

    Reserved Character              Hex Digit Pair
              ,                                        %2C
              :                                        %3A

    The URL encoded dummy URL:

    http://aws.amazon.com/onca/xml?Service=AWSDummyService&AWSAccessKeyId=00000000&Operation=DummySearch&ResponseGroup=DummyInfo%2CTopSellers%2CNewReleases&Version=2011-05-05&TimeStamp=2011-5-03T14%3A22%3A58Z

    4)  Re-arrange URL query parameters in byte order to produce the canonical string.


    Capital letters come before lower case letters. A parameter name beginning with the capital letter 'D' precedes a parameter name beginnig with 'a', the names are NOT in alphabetical order.

    Dummy URL query parameters in byte order:

    AWSAccessKeyId=00000000&Operation=DummySearch&ResponseGroup=DummyInfo%2CTopSellers%2CnewReleases&Service=AWSDummyService&TimeStamp=2011-5-03T14%3A22%3A58Z&Version=2011-05-05

    The above is the canonicalized query string used to generate a signature.

    5)  Prepend Verb, Host and Path lines (preceding canonicalized query string) with a line break.

    HTTP Verb – GET, PUT, POST, DELETE or other
    Host - aws.amazon.com
    Path - /onca/xml (if the path is empty, use a / in its place)

    Each of the verb, host and path values must be on separate lines, and they must precede the canonicalized query string.

    6)  The signing string in full.

    PUT
    aws.amazon.com
    /onca/xml
    AWSAccessKeyId=00000000&Operation=DummySearch&ResponseGroup=DummyInfo%2CTopSellers%2CnewReleases&Service=AWSDummyService&TimeStamp=2011-5-03T14%3A22%3A58Z&Version=2011-05-05

    7)  Compute digital signature using signing string, Secret Access/Signing Key and a hash algorithm.

    The encryption algorithm can be HMAC SHA1 or SHA256. Amazon provides code samples in a variety of languages available to download for their Flexible Payment Service, the samples include code for generating digital signatures, see the following link.

    http://docs.amazonwebservices.com/AmazonFPS/2008-09-17/FPSBasicGuide/index.html?APPNDX_CodeSamples.html

    For more information on the Secure Hash Algorithm (SHA) see the following link.

    http://en.wikipedia.org/wiki/SHA_hash_functions

    8)  URL encode reserved characters in the signature.

    + and = characters in the signature must be replaced by hex digit pairs.

    9)  Create a signed request by attaching the signature to the end of the request URL.


    http://aws.amazon.com/onca/xml? AWSAccessKeyId=00000000&Operation=DummySearch&ResponseGroup=DummyInfo%2CTopSellers%2CnewReleases&Service=AWSDummyService&TimeStamp=2011-5-03T14%3A22%3A58Z&Version=2011-05-05&Signature=URLencodedSignature

    The above recipe can be used for Amazon's Product Advertising API and Amazon's Flexible Payment Service. For more information on the product advertising API see this link.

    http://docs.amazonwebservices.com/AWSECommerceService/latest/DG/index.html?CHAP_Intro_AAWS.html

    Sending a signature as a URL query parameter is not the only way to sign requests, a signature can be sent instead by using the Authorization header.  See below for an explanation on how this is achieved using Amazon's S3 storage service.


    Sending signed requests to Amazon S3 storage service

    If the signature is sent as a header, it must be set out as follows:

    Authorization: AWS AWSAccesskeyId : generatedSignature

    When using an Authorization header for authentication, the format of the signing string should be as follows:

    VERB + \n

    + CONTENT-MD5 header value + \n

    + CONTENT-TYPE header value + \n

    + DATE header value + \n

    + AmzHeaders names and values in lower case and in alphabetical order + \n

    + Resource

    The format of a signing string when using query string authentication instead of an Authorization header involves a minimal change, replace the DATE header value with an Expires parameter value.  The Expires parameter value indicates when you want the signature to expire.

    The Amzheaders are any headers that start with x-amz-.

    The string to sign must be URL encoded (UTF-8 encoded).

    The content-type and content-md5 values are optional, if their values are not included in the string, newlines (\n) must still be included where the values would have been placed.

    The resource value is the bucket and key values separated by a '/ '.

    For more information on S3 signed requests see the following link:

    http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html