Cisco ACNS Software Configuration Guide for Locally Managed Deployments, Release 5.5 - Chapter 11: Configuring Content Preloading and URL Filtering on Standalone Content Engines [Cisco Application and Content Networking System (ACNS) Software]

Table Of Contents

Configuring Content Preloading and URL Filtering on Standalone Content Engines

Configuring Content Preloading for Standalone Content Engines

Support of Preloading of NTLM Authenticated Objects

Creating a Preload URL List File

Enabling and Configuring Content Preloading on Standalone Content Engines

Proxy Authentication Support for Content Preloading

Example of Configuring Proxy Authentication for Content Preloading

Stopping or Resuming Content Preloading on Standalone Content Engines

Configuring URL Filtering on Standalone Content Engines

Configuring Local List URL Filtering on Standalone Content Engines

Example of Configuring URL Filtering with Local URL Lists

Reloading Local List Files on Standalone Content Engines

Creating Custom Blocking Messages

Configuring Standalone Content Engines for N2H2 URL Filtering

Configuring Standalone Content Engines for Websense URL Filtering

About Websense Server Failover

About Websense Services

Configuring Ports for the Websense Server

Websense Issues When Upgrading to ACNS 5.4.x Software

Websense Issues When Downgrading to an Earlier Version of the ACNS Software

Example of Configuring Websense Server Failover and URL Filtering

Configuring URL Filtering with a Local Websense Server

Configuring Websense URL Filtering with External Websense Servers

Deactivating Local Websense Server Services on Standalone Content Engines

Saving Websense Configuration Files

Viewing Websense URL Filtering Statistics

Configuring URL Filtering with SmartFilter Software

About the SmartFilter Control List

About the Temporary User Override Feature

Configuring Content Engines to Bypass URL Filtering for Specific HTTP and HTTPS Requests

Displaying the Current URL Filtering Configurations

Displaying URL Filtering Statistics

Clearing URL Filtering Statistics

Configuring Content Preloading and URL Filtering on Standalone Content Engines

This chapter provides an overview of content preloading and the different types of URL filtering that are supported with standalone Content Engines that are running the ACNS 5.4.x software or later releases, and describes how to configure content preloading and URL filtering on standalone Content Engines.

This chapter contains the following sections:

•Configuring Content Preloading for Standalone Content Engines

•Configuring URL Filtering on Standalone Content Engines

•Configuring Content Engines to Bypass URL Filtering for Specific HTTP and HTTPS Requests

•Displaying the Current URL Filtering Configurations

•Displaying URL Filtering Statistics

•Clearing URL Filtering Statistics

In the ACNS 5.2.3 software and later releases, you can configure a Content Engine to monitor the performance of specific URLs. The Content Engine maintains statistics about the various response characteristics for each of the monitored URLs. For more information on this topic, see the "Monitoring Critical Disk Drives on Standalone Content Engines" section on page 21-17.

Note For complete syntax and usage information for the CLI commands used in this chapter, see the Cisco ACNS Software Command Reference, Release 5.5 publication.

For information about how to configure URL filtering on Content Engines that are registered with a Content Distribution Manager (as opposed to standalone Content Engines), see the Cisco ACNS Software Configuration Guide for Centrally Managed Deployments, Release 5.5.

Configuring Content Preloading for Standalone Content Engines

This section provides an overview of content preloading for standalone Content Engines that are running the ACNS 5.4.1 software and later releases. It also provides instructions on how to configure this feature on a standalone Content Engine.

Preloaded content is content that is retrieved and stored on a standalone Content Engine because the administrator of that Content Engine scheduled a retrieval of specific content in anticipation of user requests for that content. Content preloading is done by configuring the standalone Content Engine to create a cache request for all the content located at the origin web server that stores the primary content.

You can specify bandwidth limits for the preload process to ensure that bandwidth consumption does not exceed the specified bandwidth limits during the preload process. During the preload process, the Content Engine scans websites several link levels down for content, retrieves the specified content, and stores it locally for future requests. At a specified time, the Content Engine scans several levels of websites to verify that its content is still current, and it updates any content that has changed.

Note The ACNS 5.x software can read a file of URLs and preload the specified URL content on the standalone Content Engine. The following type of content can be preloaded on a standalone Content Engine: HTTP URLs and FTP-over-HTTP URLs. All configured HTTP and FTP-over-HTTP parameters and rules apply to the preloaded objects.

Support of Preloading of NTLM Authenticated Objects

In the ACNS 5.1.1 software and later releases, the preloading of NTLM authenticated objects is supported. This feature allows NTLM authenticated objects (authenticated objects that reside on the servers that authenticate NTLM only) to be preloaded on a Content Engine.

Note In the ACNS 5.1.1 software and later releases, NTLMv1 support of preloading NTLM authenticated objects on standalone Content Engines is available. In the ACNS 5.4.1 software release, NTLMv2 support of preloading NTLM authenticated objects was added. The ntlm version global configuration command, which was added in the ACNS 5.4.1 software release, has two command options: the version 1 option to re-enable NTLMv1, and the version 2 option to enable NTLMv2 on a standalone Content Engine. The version 1 command option is the default option.

An entry in a URL list file has the following format:

URL [depth] [domain-name:host-name:host-domain-name]

hostname and host-domain-name can be null; however, the domain name is required if NTLM credentials have been configured. (The separator is required.)
http//www.cisco.com 3 apac::
If NTLM-related information is not present in the preload URL list file entry, the authentication scheme falls back to basic authentication.

By default, the Content Engine does not cache basic and NTLM authenticated objects. To enable a standalone Content Engine to fetch specific objects and cache these objects that are authenticated with any authentication scheme (basic authentication or NTLM authentication), enter the http cache-authenticated all global configuration command.
ContentEngine(config)# http cache-authenticated all
To configure the Content Engine to cache only NTLM authenticated objects, enter the http cache-authenticated ntlm global configuration command. The cached objects are tagged as NTLM protected so that subsequent requests for these same objects are subjected to authentication before the Content Engine can serve the content to the client.

Before you preload WMT streaming media files on the Content Engine, you must enable the WMT feature on your Content Engine. If you used the Setup utility to configure WMT caching (as described in Step 15 of the "Using the Setup Utility to Configure a Basic Configuration on a Standalone Content Engine") on your Content Engine, then WMT is already enabled on the Content Engine. Otherwise, see the "Configuring WMT RTSP Streaming and Caching Services on Standalone Content Engines" section on page 9-14 for instructions on how to use the Content Engine CLI (instead of the Setup utility) to enable Windows Media services on a standalone Content Engine before you enable the preloading of Windows Media streaming files for this Content Engine.

Creating a Preload URL List File

The preload URL list file lists the URLs (HTTP or FTP-over-HTTP) to be preloaded on the Content Engine. This file is maintained by the administrator and must be created on a remote system. This file can be transferred to the standalone Content Engine for preloading access, or accessed from the remote server. The pre-load url-list-file path global configuration command specifies the path of this file.

Note In the pre-load url-list-file path global configuration command, the value for path can be a URL or a local file path.

You can place the list of URLs in a file on a local disk. You can also use the mkdir EXEC command to make a subdirectory that contains the preload URL list file. For instance, the mkdir /local1/preload-directory command creates a subdirectory called preload-directory on local disk /local1.

Each URL in the preload URL list file has an optional depth parameter. The depth parameter specifies how many levels down the preloading will be performed. For example, http://www.espn.com 3 means download http://www.espn.com and all content three levels deep. If the depth level is not specified, then the preload depth level default of 3 is used. The URLs are delimited with a carriage return as follows:
<cr>
. . . 
http://www.cnn.com 3 <cr>
ftp://ftp.lehigh.edu/ 2 <cr>
http://www.yahoo.com <cr>
. . .
<cr>
If you want to preload authenticated content to a Content Engine, the URL list file entry must be written as follows:
http://username:password@www.authenticatedsite.com/ depth level
Note In the ACNS 5.4.1 software and later releases, support for proxy authentication for preloaded content was added. For more information on this topic, see the "Proxy Authentication Support for Content Preloading" section.

When configuring a preload URL list file through the Content Engine CLI, the pre-load url-list-file global configuration command only had the HTTP or FTP option in the ACNS 5.1.x software earlier than the ACNS 5.1.5 software release. There was no method in place to fetch the preload URL list file securely.

In the ACNS 5.1.5 software release, the ability to fetch the preload URL file over HTTPS was added. If a preload URL list file contains usernames and passwords, organizations are now able to fetch the preload URL list file over HTTPS. The actual preloading of HTTPS links is not supported; only the downloading of the preload URL list file through the HTTPS protocol.

Enabling and Configuring Content Preloading on Standalone Content Engines

You can enable and configure content preloading on a standalone Content Engine from either the Content Engine GUI or the CLI.

Note From the Content Engine GUI, choose Caching > Content Preload. Use the displayed Content Preload window to enable and configure this feature on this standalone Content Engine. For more information about how to use the Content Preload window to perform this task, click the HELP button in the window.

To use the Content Engine CLI to enable and configure content preloading on a standalone Content Engine, follow these steps:

Step 1 Enable content preloading on the Content Engine.
ContentEngine(config)# pre-load enable
Step 2 Create the preload URL list file, as described in the "Creating a Preload URL List File" section.

Step 3 Specify the maximum number of concurrent requests for the URL retrieval. You can specify a value from 1 to 30 (for example, 24). The default is 10. If the number of URLs in the preload URL list file is fewer than the number of specified concurrent requests, then the lesser number is active.
ContentEngine(config)# pre-load concurrent-requests 24
Step 4 Specify the default depth level for the URL retrieval (for example, four levels deep). You can specify a value from 0 to 20. The default is 3. Setting the depth level default to 0 would be useful if you have specified URLs in preload.txt files and you do not want the Content Engine to try to preload other URLs.
ContentEngine(config)# pre-load depth-level-default 4
Step 5 Specify the path of the file that contains the URL list or a URL.
ContentEngine(config)# pre-load url-list-file path
path is the path of the file that contains the URL list or a URL. For example:
pre-load url-list-file /local1/myurllist
pre-load url-list-file ftp://ftpserver/ftpdirectory/urllist.txt
pre-load url-list-file http://server/directory/urllist.txt
Note The actual preloading of HTTPS links is not supported; only the downloading of the preload URL list file through the HTTPS protocol.

Step 6 Specify the domains to be fetched during the preload process (for example, cisco.com).
ContentEngine(config)# pre-load fetch domain cisco.com
Step 7 Specify that other domains in an HTML page should be traversed. By default, other domains in an HTML page are not traversed during the content preload.
ContentEngine(config)# pre-load traverse-other-domains
Step 8 Specify the suffixes to be excluded from the preload operation. This creates a filter for the objects that are to be excluded.
ContentEngine(config)# pre-load no-fetch suffix .mil .su .ca
Step 9 Configure a maximum bandwidth for the preloading process (for example, 50,000 kbps).
ContentEngine(config)# pre-load max-bandwidth 50000
Note With the ACNS 5.x software, you can preload WMT streaming media files that may have different bit rates at the URL specified for content preloading. You can also control WMT bandwidth using the bandwidth wmt outgoing and bandwidth incoming global configuration commands. For more information, see the "Configuring Incoming and Outgoing WMT Bandwidth and Bit Rates" section on page 9-23.

Step 10 To trigger a content preload immediately, enter the pre-load force EXEC command.

Step 11 To configure the Content Engine to preload specific content for a future time, use the pre-load schedule global configuration command. The Content Engine accesses the specified preload URL list file with a frequency set by the specified preloading schedule (set through the pre-load schedule global configuration command or the Content Engine GUI [Caching > Content Preloading]).

The default start time for the preloading operation is 00:00 (that is, the start of the day). If the end time is not specified, the preload operation is completed after all the objects have been downloaded. If you want to change this default, do the following:

a. To specify the start and end times for daily or weekly preloads, use hh:mm (where hh is the hour, and mm is the minutes, for example, 01:00). For hourly preloads, use mm to specify the start and end times. The following example shows how to specify a daily interval for scheduling the content preload. In this example, the preload operation starts every day at 1:00 AM and ends every day at 2:00 AM:
ContentEngine(config)# pre-load schedule every-day start-time 01:00 end-time 02:00
b. To specify the start time and end times for hourly preloads, the start time should be 0 and the end time should be 59. For daily and weekly preloads, the start time should be from 0 to 23, and the end time should be from 0 to 59. If the endtime option is not specified, the preload operation will continue until completion.

To configure a preload on more than one day of the week, use the pre-load schedule every-week global configuration command. The following example shows how to schedule a preload operation every week on Sunday and Wednesday from 1:00 AM to 6:00 AM:
ContentEngine#(config)# pre-load schedule every-week Sun Wed 
start-time 01:00 end-time 06:00
Step 12 Set the Type of Service (ToS) value as well as the differentiated services code point (DSCP) for all preload traffic by using the pre-load dscp global configuration command to set the Type of Service (ToS) value as well as the differentiated services code point (DSCP) for all preload traffic.

Setting the ToS or DSCP is called packet marking, allowing you to partition network data into multiple priority levels or types of service. You can set the ToS or DSCP values in IP packets based on a URL match, a file type, a domain, a destination IP address, a source IP address, or a destination port.

The ACNS 5.x software includes ToS or DSCP support for HTTP, FTP, and WMT preload traffic. Because content preloading is initiated by the Content Engine and not by the requesting client when a connection is made to an origin server, the ToS or DSCP code point on the traffic going toward the server needs to be set before contact is made with the origin server.

The following example shows how to set Type of Service support to normal:
ContentEngine(config)# pre-load set-tos normal
Note Using the pre-load dscp global configuration command takes precedence over any use of the Rules Template configuration commands involving DSCP server configurations.

Table 11-1 describes the DSCP values.

Table 11-1 DSCP Values

DSCP Value

Description

<0-63>

Valid DSCP value range

af11

Packets with AF11 dscp (001010)

af12

Packets with AF12 dscp (001110)

af13

Packets with AF13 dscp (001110)

af21

Packets with AF21 dscp (011010)

af22

Packets with AF22 dscp (010110)

af23

Packets with AF23 dscp (010110)

af31

Packets with AF31 dscp (011010)

af32

Packets with AF32 dscp (011110)

af33

Packets with AF33 dscp (011110)

af41

Packets with AF41 dscp (110010)

af42

Packets with AF42 dscp (110110)

af43

Packets with AF43 dscp (110110)

cs1

Packets with CS1 (precedence 1) dscp (001100)

cs2

Packets with CS2 (precedence 2) dscp (011000)

cs3

Packets with CS3 (precedence 3) dscp (011100)

cs4

Packets with CS4 (precedence 4) dscp (110000)

cs5

Packets with CS5 (precedence 5) dscp (101100)

cs6

Packets with CS6 (precedence 6) dscp (111000)

cs7

Packets with CS7 (precedence 7) dscp (111100)

default

Packets with default dscp (000000)

ef

Packets with EF dscp (101110)

Table 11-2 describes the TOS values.

Table 11-2 TOS Values

TOS Value

Description

<0-127>

Valid ToS value range

critical

Packets with critical precedence (110)

flash

Packets with flash precedence (48)

flash-override

Packets with flash override precedence (64)

immediate

Packets with immediate precedence (32)

internet

Packets with internetwork control precedence (96)

max-reliability

Packets with maximum reliable ToS (2)

max-throughput

Packets with maximum throughput ToS (4)

min-delay

Packets with minimum delay ToS (8)

min-monetary-cost

Packets with minimum monetary cost ToS (1)

network

Packets with network control precedence (112)

normal

Packets with normal ToS (0)

priority

Packets with priority precedence (16)

Step 13 View the status of the current preloading operation.

The following example shows the status of the current preloading operation after using the pre-load set-tos and pre-load max-bandwidth commands:
ContentEngine# show pre-load
Preloading is enabled
Number of concurrent sessions: 10
Depth level: 4
URL List File: /local1/url.txt
DSCP: set-tos normal 
Max Bandwidth: 50000 Kbps
Previous preloading operation will be continued.
Preload will not traverse other domains.
Fetch Domains: 
Fetch Suffix: 
Fetch Directory: 
No-fetch Domain: 
No-Fetch Suffix: 
No-Fetch Directory: 
Scheduling on all days
	Start Time: 00:00
	End Time : Till completion
Step 14 View the statistics associated with the current preloading, after the preload has started.
ContentEngine# show statistics pre-load
Statistics of last Preloading operation
---------------------------------------
Preloading is in progress.
List of preloaded URLs are in /local1/preload_dir/downloaded_urls.
83 objects downloaded, 2842292 bytes transferred.
Step 15 Inform your end users what the URLs of the preloaded files are so that they can use their browsers or media players to access this preloaded content.
For an example of how you can verify if the preloaded VOD files are being cached and properly distributed to clients, see the "Verifying That Preloaded VOD Files Are Cached and Properly Distributed to Windows Media Clients" section on page 9-37.

Proxy Authentication Support for Content Preloading

Typically preloading works by having a standalone Content Engine preload the content that is specified in the URL list file (the url.txt file). The URL list file also specifies the desired depth level for the URL retrieval (for example, four levels deep). The username and password are available in the URL to perform server authentication if necessary to retrieve a protected object. If there is an intermediate proxy, the preloading of content will not work because the Content Engine is unable to perform proxy authentication.

In releases prior to the ACNS 5.4.1 software release, a typical deployment was that a second Content Engine (CE2) acted as an upstream proxy server for another Content Engine (CE1). CE1 was able to support NTLM authentication if you had specified the http authentication header 401 global configuration command on CE2. In the case of NTLM authentication, requests would be authenticated (treating them as WWW-Auth), and if the credentials matched, the CE1 would retrieve the contents and the content would be preloaded on CE1.

In the ACNS 5.4.1 software release, proxy authentication support for content preloading was added. Proxy authentication support for content preloading is supported if the Content Engine has been configured to use any of the following authentication schemes for the outgoing proxy server: NTLM, LDAP, RADIUS, and TACACS+. This feature supports one level of upstream proxy server (that is, the proxy information that is obtained from the preload file only applies to the immediate upstream proxy server).

In the ACNS 5.4.1 software and later releases, if CE1 is configured to use NTLM or basic authentication as the proxy authentication scheme, CE1 is able to perform the initial proxy authentication and then retrieve the requested object from the origin server. The retrieved object can be an unauthenticated or authenticated object.

To support proxy authentication in preloading, content preloading on the Content Engine must be able to accept the proxy user and proxy domain name in order to perform proxy authentication for NTLM or Basic authentication. The following URL format for the preloading process is currently supported:

http://user1:user1@10.77.157.131/kerberos/kerberos.htm 1 acns:acns:acns

where user1 is the user name, user1 is the password, 1 is the depth level, acns is the domain name, host name, and domain in which the host resides.

The format for accepting the username, password, and domain name for proxy authentication is as follows:

proxy_user:username:proxy_pwd:pwd:proxy_domain_name:domain

The username, password, and domain name each can be up to 50 characters in length. This information must be specified in the first line of the preload list file. A colon (:) is used as separator to obtain the information from the preload list file. Consequently, do not use a colon in the password for a user.

If the username, password, and domain name for proxy authentication is not available in the preload file, then the Content Engine will use the URL format to perform the preload operation.

If the Content Engine is configured to use NTLM as the proxy authentication scheme, then the preload file would contain the following type of information:

proxy_user: user1:proxy_pwd:user1:proxy_domain_name:acns

If the Content Engine is configured to use basic authentication as the proxy authentication scheme, then the preload file would contain the following type of information:

proxy_user: tuser1:proxy_pwd:tpass1:proxy_domain_name:null

For an example of how to configure proxy authentication for content preloading on standalone Content Engines, see the next section, the "Example of Configuring Proxy Authentication for Content Preloading."

Example of Configuring Proxy Authentication for Content Preloading

In the following example, the Content Engine (CE1) is configured to have the Content Engine named CE2 be its outgoing proxy server, and is configured to use NTLM authentication to preload content:

Step 1 The Content Engine (CE1) is configured to use NTLM authentication as the proxy authentication method:
CE1(config)# ip name-server name-server
CE1(config)# ntlm server host ip-address
CE1(config)# ntlm server enable
Step 2 Clear the cache and the statistics on CE1:
CE1# clear cache
CE1# clear statistics all
Step 3 Enable the preload feature on CE1:
CE1(config)# pre-load enable 
Step 4 Configure the path for the url-list file on CE1:
CE1(config)# pre-load url-list-file /local1/preload.txt 
Step 5 Configure the outgoing proxy server (CE2) for CE1:
CE1(config)# http proxy outgoing host ip-address port-number
Step 6 On CE1, enter the following command to preserve the 407 HTTP authentication header:
CE1(config)# http proxy outgoing preserve-407
Step 7 On CE2, enable authentication:
CE2(config)# http proxy incoming port-number
Step 8 On CE1, in the first line of url-list-file, specify the credentials that are to be matched against the NTLM origin server:
CE1# type preload.txt
proxy_user:preload:proxy_pwd:preload1:proxy_domain_name:acns
http://www.yahoo.com/
http://10.77.157.60/ 
In the above preload file, the username is preload, the password is preload1, and the domain name is acns. These credentials will be sent to the NTLM origin server when CE1 receives a 407 message from the outgoing proxy server (CE2).

Step 9 Force a preload operation on CE1:
CE1# preload force
Step 10 After the forced preload operation has been completed on CE1, verify that all of the objects have been preloaded properly on CE1.

a. On CE1, check the latest_preloaded_objects files in the /local/local1/preload_dir directory.

b. On CE1, check the latest_preload_error file to verify that there are no entries indicating an error because of a proxy authentication failure or a 407 message.

c. On CE1, enter the following command to verify that the objects have been properly preloaded on CE1:
CE1# show statistics preload 
 Preloading was initiated by force. 
 Preloading started at Thu Mar 13 06:42:40 2003 
 Preloading ended at Thu Mar 13 06:42:53 2003 
 List of preloaded URLs are in /local1/preload_dir/latest_preloaded_objects. 
 Preload errlog is /local1/preload_dir/latest_preload_error. 
 Number of invalid entries in URL list file = 0 
 Total number of preloaded objects = 1 
 Total number of preloaded bytes = 570 
d. On CE2, enter the following command to verify that the user (the user with the name preload) is listed in the authentication cache on CE2:
CE2# show http-authcache
Step 11 From a web client that has CE1 as its proxy server, use the browser to issue a request for http://10.77.157.60.

Step 12 Verify that this client request is served from the cache on CE1:

a. On CE1, enter the following command:
CE1# show statistics http savings
b. Check the command output to verify that the hit counter has been incremented on CE1.

c. On CE1, enter the following command to verify that the objects have been properly preloaded on CE1:
CE1# show statistics preload 
 Preloading was initiated by force. 
 Preloading started at Thu Mar 13 06:42:40 2003 
 Preloading ended at Thu Mar 13 06:42:53 2003 
 List of preloaded URLs are in /local1/preload_dir/latest_preloaded_objects. 
 Preload errlog is /local1/preload_dir/latest_preload_error. 
 Number of invalid entries in URL list file = 0 
 Total number of preloaded objects = 1 
 Total number of preloaded bytes = 570 
d. On CE2, enter the following command to verify that the user (the user with the name preload) is listed in the authentication cache on CE2:
CE2# show http-authcache
Step 13 From a web client that has CE1 as its proxy, use the browser to issue a request for http://10.77.157.60.

Step 14 Verify that this client request is served from the cache on CE1:

a. On CE1, enter the following command:
CE1# show statistics http savings
b. Check the command output to verify that the hit counter has been incremented on CE1.
Stopping or Resuming Content Preloading on Standalone Content Engines

To stop a preload process that is currently in progress on a standalone Content Engine, use the no pre-load enable global configuration command.

If the content preloading is not completed before the scheduled end time, you can resume the preloading process to capture content using the pre-load resume global configuration command. Using this command allows you to resume downloading from the breakpoint of the previous preload, instead of starting again from the very beginning of the preload URL list file.

Note If the pre-load resume global configuration command is not set up on the Content Engine and content preloading is aborted before the scheduled end time, the next scheduled content preloading starts from the beginning of the preload URL list file.

Configuring URL Filtering on Standalone Content Engines

Some enterprises have a requirement to monitor, manage, and restrict employee access to nonbusiness and objectionable content on the Internet. Employees or students can be allowed or denied access to websites, or can be coached with information about acceptable use of the Internet. By having a URL filtering scheme on Content Engines, organizations realize an immediate return on investment as a result of increased productivity and recaptured network bandwidth, while reducing legal liability.

Table 11-3 lists the various URL filtering schemes that you can configure a standalone Content Engine to use in order to control client access to websites.

Table 11-3 URL Filtering Methods with Standalone Content Engines

URL Filtering Scheme

More Information

Local list files

Deny access to URLs specified in a list. Permit access only to URLs specified in a list. See the "Configuring Local List URL Filtering on Standalone Content Engines" section.

N2H2 external servers

Direct client requests for content to an external N2H2 server for URL filtering. See the "Configuring Standalone Content Engines for N2H2 URL Filtering" section.

Websense server

Direct client requests for content to either the local Websense plug-in or to an external Websense server for URL filtering. See the "Configuring Standalone Content Engines for Websense URL Filtering" section.

SmartFilter plug-in

Direct client requests for content to the SmartFilter plug-in for URL filtering. See the "Configuring URL Filtering with SmartFilter Software" section.

See Table B-6 for a list of the URL filtering schemes (for example, SmartFilter or Websense) that are supported for different protocols.

Although only one form of URL filtering scheme can be active at a time per protocol, many URL filtering schemes can be supported at one time. For example, if an N2H2 filter is applied to HTTP requests, then no other URL filtering scheme (for instance, Websense or SmartFilter) can be applied to this protocol. However, you could apply the local list URL filtering scheme (good lists and bad lists) to the streaming media protocols (WMT client requests and client requests over RTSP). The scheme enabled for a particular protocol is independent from that of other protocols.

Note The url-filter global configuration command takes precedence over the rule global configuration command to the extent that even the rule no-block command is executed only if the url-filter command has not blocked the request.

To ensure that URL filtering applies to every URL that passes through the Content Engine, disable all bypass features. By default, load bypass is enabled.

•To use the Content Engine GUI to disable load bypass manually, choose Caching > Bypass and then click the Load Bypass Off radio button in the Bypass window.

•To disable load bypass manually through the Content Engine CLI, use the bypass load global configuration command.
ContentEngine(config)# no bypass load enable
•To disable error handling manually through the Content Engine CLI, use the error-handling send-cache-error or error-handling reset-connection global configuration command. (By default, error handling is enabled on the Content Engine.)
ContentEngine(config)# no error-handling send-cache-error
ContentEngine(config)# no error-handling reset-connection
When both RADIUS authentication and URL filtering are enabled on the Content Engine, the user Filter-Id attribute in the RADIUS server database can be configured to bypass URL filtering.

The following example shows an example of a user Filter-Id attribute entry in the RADIUS server database:
test        Password = "test"
            Service-Type = Framed-User,
            Filter-Id = "No-Web-Blocking"
The Filter-Id attribute is defined as either No-Web-Blocking or Yes-Web-Blocking. Yes-Web-Blocking means that the request is subject to URL filtering, and No-Web-Blocking means that the request is not subject to URL filtering. If blocking is not specified, Yes-Web-Blocking is the default RADIUS filter.

Note For more information about the use of a RADIUS server for authentication purposes and URL filtering, see the "Understanding RADIUS Authentication and Authorization" section on page 17-6.

Configuring Local List URL Filtering on Standalone Content Engines

You can configure standalone Content Engines to deny client requests for URLs that are listed in a badurl.lst file, or configure them to fulfill only requests for URLs in a goodurl.lst file. The use of local list files (URL lists) applies to HTTP (HTTP, HTTPS-over-HTTP, and FTP-over-HTTP) as well as RTSP streaming media protocol. This type of URL filtering is referred to as local list URL filtering.

Tip Only one good sites file or one bad sites file can be active at one time per protocol.

The local list file for each protocol should not contain URLs that belong to other protocols. For instance, the HTTP local list file should contain only the following types of URLs: HTTP, HTTPS, or FTP URLs. In the ACNS 5.3.1 software and later releases, the WMT local list file can contain RTSP URLs.

Caution If the size of the local list file becomes too large, it can adversely effect proxy performance, because the local list file is loaded into memory when local list filtering is enabled. If the file size is larger than 5 MB, a warning message appears, but the ACNS software does not enforce size limits for the local list file. It is your responsibility to track the local list file size and ensure that it does not become so large that it degrades performance.

You can configure a standalone Content Engine to use local list URL filtering to filter the following types of client requests for content:

•Requests over HTTP (HTTP, FTP-over-HTTP, and HTTPS-over-HTTP requests)

•RealMedia requests (IETF standard RTSP protocol with RealNetworks proprietary extensions)

•WMT requests (MMS-over-HTTP and RTSP-over-RTP for Windows Media 9 clients and Windows Media 9 servers)

Filtering for native FTP and native HTTPS requests is not supported.

Support for RTSP-over-RTP (as referred to as WMT RTSP requests) for Windows Media 9 players is available in the ACNS 5.3.1 software and later releases. For WMT RTSP requests, there are three possible protocol prefixes: rtsp, rtspu, and rtspt.

If the user enters rtsp: as the protocol prefix for the URL, the Windows Media 9 player can choose to use RTSPT or RTSPU. If the rtsp bad file has a URL of rtsp://hostname/pathname and the user's URL request is rtspt://hostname/pathname, then the RTSP request from a Windows Media 9 player might get through the URL filtering. Special URL filtering for RTSP requests from Windows Media 9 players is available in the ACNS 5.3.1 software and later releases.

For WMT URL filtering, filtering for RTSP URLs (rtsp://) is only supported; there is no separate filtering support for RTSPT and RTSPU URLs. However, if you configure an RTSP URL in the badurl.lst file, then it will bock both the RTSPT and RTSPU URLs.

To use the Content Engine CLI to configure local list URL filtering on a standalone Content Engine, use the url-filter global configuration commands. In the ACNS 5.3.1 software release, the url-filter command was modified to support local list URL filtering for RTSP requests from Windows Media 9 players. In the ACNS 5.2.x software and earlier releases, the url-filter command options were as follows:
ContentEngine(config)# url-filter ?
 http For requests over HTTP
 rtsp For requests over RTSP
 wmt For WMT requests
In the ACNS 5.3.1 software and later releases, the url-filter command options are as follows:
ContentEngine(config)# url-filter ?
 http For requests over HTTP and MMS over HTTP
 rtsp For requests over RTSP - applies to real proxy, real server and cisco
 streaming engine
 wmt For WMT requests - applies to MMS and RTSP
Local list URL filtering is the only supported filtering method for WMT requests (MMS-over-HTTP and RTSP requests from WMT clients) and RTSP requests (requests from RealMedia players). The third-party URL filtering methods (N2H2, SmartFilter, and Websense software) are not supported for WMT and RTSP requests. For HTTP requests, the local list URL filtering method as well as N2H2, SmartFilter, and Websense URL filtering is supported. See Table B-4 for a list of the kinds of URL filtering that are supported for the different protocols.

Table 11-4 describes the Content Engine CLI commands for configuring a standalone Content Engine to use local list URL filtering for HTTP requests (HTTP, FTP-over-HTTP, MMS-over-HTTP, and HTTPS-over-HTTP requests).

Table 11-4 Configuring Standalone Content Engines to Use Local List URL Filtering for Requests over HTTP

CLI Command

Description

url-filter http bad-sites-deny enable

Configures the Content Engine to deny client requests to URLs in the HTTP bad site list.

url-filter http bad-sites-deny file filename

Specifies the filename of the HTTP bad site list.

url-filter http good-sites-allow enable

Configures the Content Engine to permit client requests to URLs in the HTTP good site list.

url-filter http good-sites-allow file filename

Specifies the filename of the HTTP good site list.

Table 11-5 describes the Content Engine CLI commands for configuring a standalone Content Engine to use local list URL filtering for requests over RTSP. This type of URL filtering is used with RealProxy, which is a backend RTSP server that is running on the standalone Content Engine. With registered Content Engines, RealProxy, RealSubscriber, and the Cisco Streaming Engine that are running on the registered Content Engine use this type of URL filtering.

Table 11-5 Configuring Standalone Content Engines to Use Local List URL Filtering for Requests over RTSP

CLI Command

Description

url-filter rtsp bad-sites-deny enable

Configures the Content Engine to deny client requests to URLs in the RTSP bad site list.

url-filter rtsp bad-sites-deny file filename

Specifies the filename of the RTSP bad site list.

url-filter rtsp good-sites-allow enable

Configures the Content Engine to permit client requests to URLs in the RTSP good site list.

url-filter rtsp good-sites-allow file filename

Specifies the filename of the RTSP good site list.

Table 11-6 describes the Content Engine CLI commands for configuring a standalone Content Engine to use local list URL filtering for WMT requests (WMT requests over RTSP). If you configure an RTSP URL in a WMT bad site list, then it blocks both the RTSPT and RTSPU URLs as well as the RTSP URL that is specified in the bad site list.

Table 11-6 Configuring Standalone Content Engines to Use Local List URL Filtering for WMT Requests

CLI Command

Description

url-filter wmt bad-sites-deny enable

Configures the Content Engine to deny client requests to URLs in the WMT bad site list.

url-filter wmt bad-sites-deny file filename

Specifies the filename of the WMT bad site list.

url-filter wmt good-sites-allow enable

Configures the Content Engine to permit client requests to URLs in the WMT good site list.

url-filter wmt good-sites-allow file filename

Specifies the filename of the WMT good site list.

In the ACNS 5.3.1 software and later releases, the url-filter wmt global configuration commands apply to RTSP. URL filtering for RTSP requests is used when the client is a Windows Media 9 player and the server is a Windows Media 9 server. If an earlier version of the Windows Media player is used (for example, Windows Media 7 players), the MMS-over-HTTP protocol is used instead of the RTSP protocol to service the content request from the Windows Media player.

Example of Configuring URL Filtering with Local URL Lists

To configure a standalone Content Engine to use a local list file to deny client requests for specific HTTP URLs, follow these steps:

Step 1 Create a plain text file named badurl.lst.

In this file, enter the URLs that you want to block. The list of URLs in the badurl.lst file must be written in the form http://www.domain.com/ and be delimited with carriage returns.

Step 2 Copy the badurl.lst file to the /local1 system file system (sysfs) directory of the standalone Content Engine.

Tip We recommend creating a separate directory under local1 to hold the bad lists, for example, /local1/filtered_urls.

Step 3 Point the Content Engine to the bad URL list.
ContentEngine(config)# url-filter http bad-sites-deny file local/local1/badurl.lst
Step 4 Configure the Content Engine to actively deny the URLs.
ContentEngine(config)# url-filter http bad-sites-deny enable 
Step 5 Reload the new bad site list on the standalone Content Engine.
ContentEngine# url-filter local-list-reload http
To configure a standalone Content Engine to use a local list file permit specific HTTP URLs to the exclusion of all other URLs, follow these steps:

Step 1 Create a plain text file named goodurl.lst.

In this file, enter the URLs that you want to exclusively allow. The list of URLs in the goodurl.lst file must be written in the form http://www.domain.com and be delimited with carriage returns.

Step 2 Copy the goodurl.lst file to the /local1 sysfs directory of the Content Engine.

Tip We recommend creating a separate directory under local1 to hold the good lists, for example, /local1/filtered_urls.

Step 3 Point the Content Engine to the goodurl.lst file.
ContentEngine(config)# url-filter http good-sites-allow file local/local1/goodurl.lst
Step 4 Configure the Content Engine to actively permit only the good URLs.
ContentEngine(config)# url-filter http good-sites-allow enable
Step 5 Reload the new good site list on the standalone Content Engine.
ContentEngine# url-filter local-list-reload http
Reloading Local List Files on Standalone Content Engines

When you update the badurl.lst or goodurl.lst file, use the url-filter local-list-reload EXEC command to reload the good site or bad site lists on the standalone Content Engine if the URL list feature is enabled.

url-filter local-list-reload {http | rtsp | wmt}

The syntax is as follows:

•http reloads the new local lists for HTTP requests (HTTP, FTP-over-HTTP, MMS-over-HTTP, and HTTPS-over-HTTP requests).

•rtsp reloads the local lists for requests over RTSP (requests from RealMedia clients).

•wmt reloads the local lists for WMT requests (RTSP-over-RTP [the standard IETF RTSP protocol with Microsoft proprietary extensions] requests from Windows Media 9 players).

The following examples shows how to reload new good site or bad site lists on a standalone Content Engine:
ContentEngine# url-filter local-list-reload http
ContentEngine# url-filter local-list-reload rtsp
ContentEngine# url-filter local-list-reload wmt
Creating Custom Blocking Messages

In the case of local list URL filtering, you can configure standalone Content Engines to return a customized blocking message to the client that requested content that is served through the Content Engine. The custom message must be an administrator-created HTML page named block.html. Make sure to copy all embedded graphics associated with the custom message HTML page to the same directory that contains the block.html file. The following is an example of the contents of the block.html file:
<TITLE>Cisco Content Engine example customized message for url-filtering</TITLE>

<H1>
<CENTER><BLINK>
P
R
A
D
E
E
P
'
S
</BLINK>
Blocked Page
</CENTER>
</H1>


<IMG src="/content/engine/blocking/url/my.gif">

This page is blocked by the Content Engine.

If the block.html file is updated, it will automatically display its new message without requiring you to reenter the url-filter http custom-message command.

In the following example, a block.html file displays this custom message when the standalone Content Engine intercepts a request to the blocked site:
This page is blocked by the Content Engine
In the block.html file, objects (such as .gif, .jpeg, and so on) must be referenced within the custom message directory string /content/engine/blocking/url, as shown in the preceding example.

To enable the customized blocking message, use the url-filter http custom-message global configuration command and specify the directory name. To disable the custom message, use the no url-filter http custom-message command.

You can enable and disable the url-filter http custom-message command without affecting the good-sites-allow and bad-sites-deny configuration.

Note Do not use local1 or local2 as directories for custom blocking messages. Create a separate directory under local1 or local2 for holding the custom message file.

Contact your administrator if you have any questions concerning access to the blocked site you requested.

Configuring Standalone Content Engines for N2H2 URL Filtering

N2H2 is a globally deployed URL-filtering solution that can filter HTTP, FTP, or HTTPS requests based on destination hostname, destination IP address, and username and password. N2H2 relies on a sophisticated URL database exceeding 15 million sites and is organized into over 40 categories using both Internet technology and human review. See http://www.n2h2.com for further information on N2H2 filtering products.

Note See Table B-4 for a list of the kinds of URL filtering that are supported for the different protocols with an N2H2 server.

N2H2 supports three filtering methods. Table 11-7 lists the N2H2 features supported by the Content Engine. One N2H2 server can support multiple Content Engines simultaneously.

Table 11-7 Supported N2H2 Features

N2H2 Feature

Description

Global filtering

Applies filtering to all HTTP requests (HTTP, FTP-over-HTTP, or HTTPS-over HTTP requests.)

User-based filtering

Applies filtering to specific users or groups.

Client IP-based filtering

Applies filtering to specific client IP addresses.

Transparent authentication

Performs transparent authentication by passing back the initial response header to the client using the HTML page in IFP responses.

Standalone Content Engines can use an N2H2 enterprise server as a filtering engine and enforce the filtering policy configured on the N2H2 server. (See Figure 11-1.) The standalone Content Engine and the N2H2 server use Internet Filtering Protocol (IFP) Version 2 to communicate with each other. When the Content Engine receives a URL request, it sends an IFP request to the N2H2 server with the requested URL. The N2H2 server does some necessary lookups for the URL and sends back an IFP response. Based on the N2H2 server's IFP response, the Content Engine either blocks the HTTP request by redirecting the browser to a page where a blocking message is displayed or proceeds with normal HTTP processing by sending the URL request to an origin server.

Figure 11-1 N2H2 Filtering

Note URL filtering using an N2H2 server is applied to HTTP traffic (HTTP, FTP-over-HTTP, or HTTPS-over-HTTP requests) before the Rules Template is applied, regardless of whether the requested object is in the