1 Reply by Stuart (edited by Stuart 13-10-2008 09:19:22)

  • Stuart
  • Guest

Re: Digest authentication with PHP 4 cracked at last!

One of the things I have wanted to do for some time on the Hoverfly Recording Scheme (www.hoverfly.org.uk) and Dipterists Forum (www.dipteristsforum.org.uk) web-sites is to provide access to lists of species that have been recorded from a given site (e.g. SSSI, LNR). BUT to actually produce any results, this requires "download access" to the datasets involved. Very few datasets give public download access, so the results are disappointing!

The way round this is to use the secure web-services and logon behind the scenes as a user who does have the necessary access to the datasets I am most interested in (the ones managed by Dipterists Forum). So I have been trying, on and off, to get digest authentication via NuSoap working.

Some useful information was provided by Dan Jones and Andy Brewer. Dan Jones' code (with the URLs updated from the old numeric addresses) is:

$headers = http_parse_message(http_head('http://www.nbnws.net/ws/secure/protected.jsp'));
$headers = $headers->headers;
$wauth = $headers['Www-Authenticate'][2];
$wauth = preg_replace("/Digest /","",$wauth);
$wauthkeys = split(", ",$wauth);
$dgheaders = array();
foreach($wauthkeys as $seg) {
   list($key,$val) = split("=",$seg);
   $val = preg_replace("/\"/","",$val);
   $dgheaders[$key] = $val;
}
print "<h2>headers</h2>";
print_r($dgheaders);
$client = new nusoapclient('http://www.nbnws.net/ws/WSDL', 'wsdl'); 
 
$client->setCredentials($user,$password,'digest',$dgheaders);

Unfortunately, this uses PHP 5 extensions (http_parse_message, http_head, built in SOAP support) which are not available in PHP 4 - and my ISP does not support PHP 5. So I have been trying to find a way of achieving the same thing in PHP 4 (to be exact: version 4.0.4 onwards).

Anyway, to cut to the chase, below are the code and some explanation.

2 Reply by Stuart (edited by Stuart 13-10-2008 07:55:00)

  • Stuart
  • Guest

Re: Digest authentication with PHP 4 cracked at last!

require 'nusoap/nusoap.php';

error_reporting(E_ALL ^ E_WARNING);

$user = '<username>';
$pswd = '<password>';
$url  = 'http://www.nbnws.net/ws/secure/protected.jsp';
$wsdl = 'http://www.nbnws.net/ws/secure/WSDL';

    // Try to open the URL (read-only) - we are expecting this to fail because it is in the secured area
    $fp = fopen($url, 'r');
     
    // Check response header from the server
    if (isset($http_response_header))
    {
        // is it the 401 error we are expecting?
        if (strncasecmp($http_response_header[0], 'HTTP/1.1 401', 12) == 0)
        {
            // recover the digest parameter string
            for ($i = count( $http_response_header ) - 1; $i >= 0; $i-- )
            {
                if (strncasecmp('WWW-Authenticate', $http_response_header[$i], 16 ) == 0 )
                {
                    $wauth = $http_response_header[$i];
                    break;
                }
            }
            // did we find it?
            if (isset($wauth))
            {
                // clean out the start of the string which we don't need
                $wauth = preg_replace("/WWW-Authenticate: Digest /", '', $wauth);
                // split it up on commas
                $keys = split(', ', $wauth);
                // recover the parameters into an array to use in setCredentials
                $dparams = array();
                foreach($keys as $k) 
                {
                    // split into a key,value pair on =
                    list($key, $val) = split('=', $k);
                    // clean out quotes from $val
                    $val = preg_replace("/\"/", '', $val);
                    $dparams[$key] = $val;
                }
        
                //NB: no proxy settings! - these are the four blank parameters if you need them
                $client = new soapclient($wsdl, 'true', '', '', '', '');
                $client->setCredentials($user, $pswd, 'digest', $dparams);
            }
            else
                die( 'Error connecting: Authentication information from the server not found.');
        }
        else 
            die( 'Error connecting: ' . $http_response_header[0]);
    }
    else
        die( 'Error connecting: Bad URL, timeout, etc.');

3 Reply by Stuart (edited by Stuart 13-10-2008 09:29:23)

  • Stuart
  • Guest

Re: Digest authentication with PHP 4 cracked at last!

Explanation: Background to HTTP Authentication

Scenario: The server has some resources that are restricted and should only be served up to authorised users. These are kept in a specific area of the server known as a "realm" (physically, this may be the files on a particular directory and its sub-directories). Authorised users are identified (Authenticated) by a username and password. We are not worried here, how the username and password are created or stored - just that they exist and are known to the server.

The HTTP 1.1 protocol has built in mechanisms to support this.

The simplest is "Basic authentication". The way this works is:
- the client (your web-browser) requests a resource that is located in the protected realm
- the server sees that this resource requires basic authentication and sends back an error "HTTP Error 401 - Unauthorized". The HTTP header of this response tells the client that basic authentication is required - i.e. the server wants a username and password.
- the client prompts the user for their username and password (the ubiquitous login dialog box!) and then re-sends the request, this time including the username and password encoded in the HTTP header
- the server checks that the username and password are valid and serves up the requested resource.

This is pretty straightforward and easy to understand (and very widely used), but its drawback is that the username and password are sent over the internet in the HTTP header as plain text. So anyone who can access the stream of messages going backwards and forwards (e.g. someone using a packet sniffer at some intermediate internet node) can easily get your username and password.

The system used by the NBN Gateway is "Digest authentication". It is a bit more secure, but more complicated. It provides a way of telling the server that the client knows the user's password without having to actually send it. The trick is that the server sends the client a token called a "nonce" ("n once" - a number that is valid for one time only) and the username, password and nonce (and some other parameters) are combined using a one-way hashing algorithm (md5). It is the result of this md5 hash that are sent to the server. The server can do exactly the same calculation at its end and, if the answers agree, then both client and server must be using the same username/password combination and the server can authenticate the user.

So the process is:
- the client (your web-browser) requests a resource that is located in the protected realm
- the server sees that this resource requires digest authentication and sends back an error "HTTP Error 401 - Unauthorized". The HTTP header of this response tells the client that digest authentication is required and includes the necessary parameters (nonce, etc).
- the client prompts the user for their username and password and combines them with the nonce and other parameters recovered from the HTTP header to calculate the md5 hash.
- the client re-sends the request, this time including the username and md5 hash in the HTTP header
- the server reads the username, accesses its own copy of that user's password and calculates the md5 hash using the nonce and other parameters it previously sent. If the results are the same as those passed back by the client, the user is authorised and it can serve up the requested resource.

The advantage of this is that the password is not sent. Somebody who can access the message stream can see the username and hash, but the nature of the hashing process makes it impossible to recover the password from it. The nonce is also designed to have a limited lifetime so that the particular hash is only valid for a limited time (typically some number of seconds). This protects you password, but is still not particularly secure because the protected resource is returned as plain text. So the snooper in the middle still gets to see it, even if they cannot get your password!

The NuSoap library includes a method to allow for various types of authentication: setCredentials(username, password, authentication_type, parameters)

- username and password are strings. Note that these are case sensitive! This is not normally the case when you login to the NBN Gateway, but IT IS the case in this context. They must match EXACTLY the Gateway's version.

- authentication_type is a string which can be "basic" or "digest" (or other options we don't need to worry about)

- if authentication_type is "digest", then parameters is an associative array with the fields realm, nonce, qop and opaque which are recovered from the 401 error message's HTTP header returned by the server.

So, what we need to do to achieve a NuSoap connection to secured NBN web-services is:
- request some resource on the NBN Gateway server's secured realm
- get back a 401 error and recover the necessary parameters from it
- instantiate our NuSoap client, passing it the URL of the NBN Gateway's secure WSDL
- call its setCredentials method with a valid username and password and with the parameters recovered from the 401 error header

We can then use the client object to make web-service requests as usual. Because we asked for the secured version of the WSDL, these will be the secured versions of the web-service calls. NuSoap will handle the digest authentication dialog with the server.

Just for the sake of completeness, if we want to be really secure we would need to use the HTTPS/SSL (Secure Socket Layers) system in which all the messages passed backwards and forwards are encrypted. NuSoap can supports this - the third, authentication_type, parameter is "certificate". This is not supported by the NBN Gateway and is outside the scope of what we are doing here, but the downside is that everything has to be encrypted before it is sent and decrypted when it is received, so there is a considerable overhead on both server and client. So it should only be used when REALLY necessary (like sending your credit card details!).

4 Reply by Stuart (edited by Stuart 13-10-2008 08:08:18)

  • Stuart
  • Guest

Re: Digest authentication with PHP 4 cracked at last!

Explanation: The code

Andy Brewer supplied the URL for a suitable protected resource that can be requested to kick this process off 

http://www.nbnws.net/ws/secure/protected.jsp

Dan Jones' code to request this and obtain the header returned by the server is:

$headers = http_parse_message(http_head('http://www.nbnws.net/ws/secure/protected.jsp'));

But unfortunately this uses PHP5 specific features and does not work in PHP4.

The PHP 4 code I have come up with to do the same job is:

$fp = fopen('http://www.nbnws.net/ws/secure/protected.jsp', 'r');

The "r" parameter indicates it should be opened for "read-only". The system variable $http_response_header can be used to access the header data from the response.

The header should be something like this:

Server wrote:

HTTP/1.1 401 Unauthorized
Server: Apache-Coyote/1.1
Pragma: No-cache
Cache-Control: no-cache
Expires: Thu, 01 Jan 1970 01:00:00 GMT
WWW-Authenticate: Digest realm="Example Digest-Based Authentication Area", qop="auth", nonce="4d4c36dc5e31dea5f3ddac21aa938b20", opaque="965bf9f16bfb24fe8ab9653aa6a2fe60"
Content-Type: text/html;charset=utf-8
Content-Length: 954
Date: Fri, 10 Oct 2008 16:04:43 GMT
Connection: close

Note that you will also get a warning from PHP (because HTTP returns an error) which can show up as a message in your generated web-page:

Server wrote:

Warning: fopen(http://www.nbnws.net/ws/secure/protected.jsp): failed to open stream: HTTP request failed! HTTP/1.1 401 Unauthorized in test.php on line nn

This can be suppressed by including the following code somewhere in your program. This just turns reporting of warnings off, but leaves all other error reporting unaffected:

error_reporting(E_ALL ^ E_WARNING);

We are interested the 6th line of the HTTP header "WWW-Authenticate: Digest ..." which contains the parameters we need.

So, first I check that we got the header and it is the 401 error I was expecting and throw an error if it is not:

    // Check response header from the server
    if (isset($http_response_header))
    {
        // is it the 401 error we are expecting?
        if (strncasecmp($http_response_header[0], 'HTTP/1.1 401', 12) == 0)
        {
             ...
        }
        else 
            die( 'Error connecting: ' . $http_response_header[0]);
    }
    else
        die( 'Error connecting: Bad URL, timeout, etc.');

and then extract the parameters from $http_response_header into an associative array $dparams. Once that is done, we can instantiate the NuSoap client object and call its setCredentials method:

        $client = new soapclient($wsdl, 'true', '', '', '', '');
        $client->setCredentials($user, $pswd, 'digest', $dparams);

One other thing to note: In the various NuSoap examples I have found, it is normal to test whether we successfully got a client object after we have instantiated it:

        $client = new soapclient($wsdl, 'true', '', '', '', '');
        // check we got a connection
        $err = $client->getError();
        if ($err) 
        {
            die( 'Error connecting: ' . $err);
        }

Don't do that just here! You WILL get an error reported:

Server wrote:

Error connecting: wsdl error: HTTP ERROR: Too many tries to get an OK response

I guess that this is because the authentication with the server has not yet taken place.

5 Reply by andyb

  • andyb
  • Member
  • Offline
  • Registered: 01-09-2006
  • Posts: 22

Re: Digest authentication with PHP 4 cracked at last!

Hi Stuart,
Well done getting all that working and thanks for the detailed explanation. At last, we have examples of working authenticated clients for the three major platforms.
Andy