Protecting your Information

When considering restricting access to pages on the Web, there are really two issues. One concerns allowing access to certain visitors, or groups of visitors, to specified pages, the other relates to restricting access to the indexing robots, also known as 'spiders' to prevent certain pages from being listed in search results.

Access Authentication

Where it is required to only allow access to a group of people, this can be achieved using a technique called Access Authentication. This requires a visitor to 'log on' to the page by giving a pre-determined user name and password combination before being allowed access.

For example, this page is password protected. If you follow the link, use the user name guest and the password password.

How Secure Is It?

In Basic HTTP Authentication, the password is passed over the network not encrypted but not as plain text - it is uuencoded. Anyone watching packet traffic on the network will not see the password in the clear, but the password will be easily decoded by anyone who happens to catch the right network packet. So basically this method of authentication is roughly as safe as telnet-style username and password security - if you trust your machine to be on the Internet, open to attempts to telnet in by anyone who wants to try, then you have no reason not to trust this method also.

In MD5 Message Digest Authentication, the password is not passed over the network at all. Instead, a series of numbers is generated based on the password and other information about the request, and these numbers are then hashed using MD5. The resulting digest is then sent over the network, and it is combined with other items on the server to test against the saved digest on the server. This method is more secure over the network, but it has a penalty. The comparison digest on the server must be stored in a fashion that it is retrievable. Basic Authentication stores the password using the one way crypt() function. When the password comes across, the server uudecodes it and then crypts it to check against the stored value. There is no way to get the password from the crypted value. In MD5, you need the information that is stored, so you can't use a one way hashing function to store it. This means that MD5 requires more rigorous security on the server machine. It is possible, but non-trivial, to implement this type of security under the UnixTM security model.

Robot Exclusion Standard

It is sometimes desirable to prevent a page on the WWW, or a group of pages, from being indexed by the various search tools. For example, this is commonly required if a site is being developed by a team, and not yet ready for release.

Most, if not all, search tools run programs (referred to as wandering robots, or spiders). Using the Robot Exclusion Standard it is possible to specify directories on the server which these programs are forbidden to enter (of course, this only works with compliant spiders).

The spiders can be "asked" not to index an individual page by using the HTML META element: <meta name="robots" content="noindex,nofollow">.

[ Design | Services | Clients | Contact TNE | Information | Search TNE ]