The following subjects will be covered in this talk:

This shows the MDS architecture for network monitoring. The top left box represents those programs present on a monitoring machine; the top right box the elements of the MDS server (which may or may not be the same machine as before - we recommend not); the bottom right hand box is the network index server.

On the monitoring machine resides a normal EDG WP7 network monitoring tool, such as PingER, IperfER or UDPMon. These tools consist of scripts which are run periodically by Cron that make measurements. The results of the measurements are stored in flat text files. A web interface using CGI perl-scripts can be used to view the results of past measurements and draw graphs of those measurements.

The MDS information provider queries PingER, IperfER or UDPMon web interfaces to extract the most recent measurements. The results are converted to an LDAP tree and returned to the MDS program. The information provider scripts are called when the information stored within the MDS becomes stale.

The network information index contains all the mappings between known CEs and SEs and their closest network monitoring machines. This information can then be used to make estimates of measurements between pairs of CEs and SEs.

R-GMA is being used to replace the Globus MDS. Therefore, the LDAP/MDS information providers will be phased out and will be removed in EDG Testbed 2.

Work on publication of network monitoring information with R-GMA was nominally started near the beginning of WP3's work on R-GMA.

The R-GMA information providers for active network measurements will be deployed with the next full EDG release (EDG 2) and will be deployed at as many sites as possible. The network monitoring information will be used by a network cost function which itself will be used in data replication decisions.

The network monitoring architecture consists of three main parts: the R-GMA framework, network monitoring producers and an archiver

R-GMA framework components

The R-GMA framework consists of the producer and consumer servlets, the registry/arbitrator and the configuration of datagrid nodes. These parts are outside the control of WP7, but are guaranteed to be available - the internal workings of the R-GMA servlets should be completely transparent to the user.

Network monitoring producers

The network monitoring producer itself consists of three parts.

The monitoring tools such as PingER, IperfER and UDPMon are the first part. They take measurements, and store their results in an internal database as normal and present information via a web interface, as normal. The tools have been modified to talk to a FIFO (or "named pipe") in the format expected by the netmon2rgma daemon.

The netmon2rgma daemon listens on that named pipe for messages indicating that a measurement has been made. The measuring tools used by WP7 (PingER, IperfER and UDPMon) all use scripts which are run periodically by cron, and so R-GMA producer information cannot be stored between invocations. The netmon2rgma daemon handles all the producers for these tools, and publishes results from them using the appropriate producers.

The netmon2rgma daemon uses the Perl R-GMA Producer API, and talks to the Producer Servlet. That servlet need not be on the same machine as the API, indeed it is advantageous for it not to, as the network measurement nodes should be loaded (processor and network use) as little as possible.

Archiver

One of the tools provided by WP3 is an archiver servlet. This servlet populates a database based upon the queries it has been provided. Streaming consumers extract data from any producer which matches the query, the database is populated and the data re-published using a database producer. The WP7 network monitoring archiver saves all of the network monitoring information produced on the grid, as well as the joining tables which link between computing/storage elements and the closest network monitoring hosts. The database producer allows R-GMA queries to the database, and it is possible to perform complicated join queries on the database (using the private registry API).

The GridFTP architecture is very similar to the general monitoring architecture. In fact, the archiver and registry would be the self-same programs running on the same servers. The GridFTP server daemon generates a logfile in /var/log/gsiwuftpd.log, and this is polled frequently by the gridftplog2rgma daemon. In this case, the gridftplog2rgma daemon is persistent, and so it can handle its own producers without the aid of netmon2rgma.

As before, the location of the producer servlet is not really important to us. For GridFTP, running on a Storage Element or Computing Element, we have no special preferences as to the location of servlets.

The following packages are available for download from the EDG package repository: