RETS Modules

Here are the modules, along with short descriptions that I have settled on for this application.

• retsAuth - Handles basic authentication to the RETS server including logging in and out. Stores the loginResponse.xml file. Stores the login cookie that is required with each request.
• retsMetadata - Fetches the metadata files from the server. Parses each to discover additional metadata further down the hierarchy, then repeats. Saves all files local with a standardized naming convention.
  
• retsMetadataStore - Traverses local metadata files, parses them and stores them in the database.
  
• retsSchema - Using the metadata that is stored in the DB, creates the database tables and relationships that will be required to store the data that will be pulled from the RETS server.
• retsDataSearch - Queries the RETS server and saves the returned XML.
  
• retsDataStore - Parses and stores returned XML files in the DB.
• retsObject - Grabs RETS objects from the server such as images and videos. Stores them locally. Updates lookup tables in the DB.
• retsController - Handles sequencing the processes. Checks local metadata versions against current to trigger updates to the schema. Runs count queries to insure local DB is in sync with the RETS server.

The breakdown should clue you in that I have decided not to leave the RETS data in its native format (XML) for accessing. I did quite a bit of experimenting with loading the metadata into memory and querying against it with xpath. While it does work, you can easily wind up with 4 or 5 GB of metadata loaded into memory, um, no thanks. I also created a xmlLoader.cfc that would dynamically load the metadata files into memory only when they were called, but you then have to run cleanup routines to clear them out. In the end it just did not seem worth it to me when I could just parse it all into the DB and get blazing fast performance. I will toss up a DB diagram for the metadata schema when I get a chance.