Davin's test page

Davin, you can do anything you want with this page. Anyone else reading this page may wish to skip the details and go to the last horizontal line where we discuss that it might be time for Barbara Dunbar to begin helping in the maintenance of the website.

This reports on a test of the "USMS_FTP" facility developed to make possible the transfer of SwimGold.org from an NT Server to a Unix server and to continue maintenance in the future with ease similar to the ease by which it was administered on the NT Server.

Critical to maintenance of SwimGold is a facile way to synchronize resources on the home computer with the server. On the NT Server, this was done with a facility written by Carl House and Fred Waid which used an HTTP protocol dependent on software resident on the NT Server. To aid in the transfer to the USMS server and to replicate the synchronization facility, Davin Church was asked to help. The transfer took 3 days, including modification of all files for the fact that the NT Server was not case sensitive whereas the Unix server is case sensitive. (Case insensitivity has certain advantages which we did use.) CuteFTP-Pro (version 2.0) was used during this process, especially whenever one wanted to make a visual inspection of the server. But, most work was done by Davin's FTP facility. Here is a report on an experience on the evening of April 24, early evening of the third day.

A large portion of SwimGold was recompiled (qkw_Master) to solve problems of case sensitivity. The FTP facility was started at 4:11 pm. As it starts, it must analyze 95 folders in SwimGold to determine which files need to be uploaded. This takes about 3 minutes. On completion of that, it reported that 3058 pages needed to be uploaded and began uploading them. Some interruption occured at 4:48 pm when 457 files remained to be uploaded. (2601 files in 37 minutes means each file took about .85 seconds to upload.) The hung condition remained while Carl ran an errand. At 5:40 he returned and noticed that it had not resumed upload (one socket was still active). So, the upload was aborted at 5:40 and restarted at 5:41. The FTP facility examined the 95 folders again and concluded that 478 needed to be uploaded (some were revised by Carl while the earlier upload was underway). The upload of the 478 files began at 5:44pm and ended at 5:57pm (1.63 seconds per file). No error messages were reported in either the first run nor the second. The only human action needed was to start the process. No settings or adjustments were needed; one click started it. It was evident that the process can run in background and can be initiated by the Windows event manager so that Barbara Dunbar, for example, does not even need to think about uploading. Synchronization can occur at whatever frequency we desire, and can be manually started if desired.

Another test (8:20pm, 4/24/02) occured with the recompiling of the major Top Ten datapages (TTDBEXPORT). \tt\age has 919 data files, \tt\lmsc\ has 505 data files and \tt\det\ has 1514. These 2938 files were recompiled in 24 minutes (.5 seconds per htm file). Then the FTP facility was started to upload them. The FTP facility uploaded 2968 files in 50 minutes (1 second per file).

This particular file has been continually modified during these experiments. Since only 1 file was being changed, CuteFTP-Pro was thought to be the logical vehicle for upload. So, the appropriate subdirectories were carefully aligned in the left and right panes, and the single file desired for upload was clicked, a "careful and intelligent command". However, the "folder synchronization" rules overrode this "careful and intelligent command" and the revised file was not uploaded. (After considerable effort, it seemed that the only way to get around the "synchronization" rules was to delete the file on the server. Then Cute-FTP Pro did upload the file.) These "folder synchronization" commands can't be handled casually. They must evolve to be trustworthy, dependable, easy and stable (rarely if ever changed). In this case, after several time wasting attempts to upload the single file using CuteFTP-Pro, Davin's FTP facility was used. It took it's 3 minutes to analyze the 12,000 files, identifying 34 files needing upload, and it uploaded them in 30 seconds or so. Still, it failed to upload the "temp.htm" file because the server time stamp was 2 minutes later than the save date on the home computer. Then these last notes were made, the local time stamp became later than the server, CuteFTP-Pro allowed it to be uploaded, and it was. Davin's FTP facility would also have uploaded it, but CuteFTP-Pro was already loaded to the proper folders (on both sides) and was easier, so it was used. The point of this is that great care is required to evolve good criteria for what should be uploaded and what should not in our scheduled synchronization procedures. The idea of articulating criteria each time one loads the software is very counter-productive. We need processes that are dependable and that do not require exceptional care on the part of system users who really want to make rapid progress in easy to use systems. With this last change, another attempt to upload was tried, and it failed as the "folder synchronization" rules overrode the users "careful and intelligent command" because the clock in the server was 5 minutes faster than the clock in the home computer. We may need to start testing file size as a criteria and we may need to consider a checksum.

We are imagining that we can begin this process of keeping everything on the server and also synchronized on both Barbara's and Carl's computers now with the facility that Davin has provided. We don't have a user interface yet, so Davin can write what is needed to install the automated FTP facility in Barbara's computer with minimal ability for her to monitor its activity and start it out of its normal routine if she wishes. Carl will also have the facility and will be refining its use. This is all Barbara needs to take full control of the narrative text files (.txt) allowing Carl to simply use what Barbara maintains. Barbara will need to select an editor and she and Carl will have to work out how she should use it. She will not have the ability to compile webpages until we have a proper user interface. But, this is a good first step at sharing the technical work with another Committee member.

The following principles would apply to installation of this in Barbara's computer (if we are to do it). At present we will do the least possible because the database project is not complete and we have very limited funds. We will avoid anything that has high likelihood of need for support before the RFP.

Structure on the server will be same as now, no revision at this time.
No automatic actions. Synchronization only occurs when Barbara causes it.
No protection against Carl & Barbara walking on each other's work (except their agreements).
No install procedure. Simply copy required files into her computer and create an icon on her desktop.
Clicking on that icon will provide several choices.
1. Upload (text files changed so they will be integrated into Carl's work)
2. Download (any files on server that have been uploaded by Carl)
3. Both Upload & Download (obviously only files that have changed will be copied)
She will only work on text files for now (.txt files in qksource folders).
The FTP facility will not enable Barbara to create a new folder. She will use Windows Explorer to do that.
Choice of an editor for Barbara and how much HTML she learns is completely outside the scope of Davin's involvement.
The facility will be called TTDB_FTP (acronym for Top Ten DataBase).
We will install it on Barbara's computer in early May when her new computer is delivered.
Carl will need to be able to modify the facility to make it work best for Barbara. For example, if her internet access is slow we might want to give her the choice of uploading 1 folder only.
A record of downloads and uploads will be accumulated in a text file for examination in any editor.