Fulford Consulting Ltd.

APP Flare support & training.

0793 572 8612.

Flare help pages.

Loading LLPG extracts into Authority/Flare.

Synopsis

The LLPG team extracts modifications from the LLPG database each week as "dtf" files. These are downloaded to the Authority/Flare server and then imported into Authority/Flare using "ADL". Special scripts used to pre-process the files ensure that files are loaded in the correct order, and that duplication is avoided.

Receiving LLPG extracts from the LLPG team.

Extracts are sent by e-mail each week to the Environment & Public Protection ICT Manager. It is essential that all extracts are loaded in the order of production. Arrangements must be made to ensure that if the post holder changes the continuity of updates is maintained.

The extracts are are supplied with names in the form:

    DddmmyyThhmmss.csv

where

D
Literal constant character "D"
dd
2 digits representing the day of the month
mm
2 digits representing the month of the year
yy
2 digits representing the the year.
T
A literal upper case character T.
hh
2 digits representing the hour of completion.
mm
2 digits representing the minutes past the hour at the time of completion.
ss
2 digits representing the seconds past the minute at the time of completion.
csv
Constant literals indicating a file with comma separated values

eg. D200804T074218.csv

The files need to be transferred to the directory /home/llpg/extracts on the Authority/Flare server.

Downloading extracts to the Authority/Flare server.

The extract files are saved to the flare "tools" drive in llpg directory ie.

     //flare/tools/llpg/extracts
  

It might be preferable if the files were e-mailed directly to the Authority/Flare server and processed by procmail to extract them to the correct directory.

Renaming extracts with "sort_extracts".

Once the files have been transferred they need to be renamed to ensure that utilities such as 'ls' will list them in date order. If multiple files are being uploaded then this process is essential. Failure to load the files in the correct order will corrupt the data.

Access the flare server is available to authorised systems administrators via secure shell (ssh). No insecure channels or generic logins are permitted. All administrators must login in with their own login id and then use sudo to switch to root or other privileged accounts.

Change the current working directory to /home/llpg. Execute the command ./sort_extracts. This will run a local script which takes files found in ./extract and creates renamed copies in the directory ./sorted

Duplicate extracts.

If a file of the same name already exists in ./archive sort_extracts will write an error message to screen asking the user to check the extracts. A copy will not be made to the sorted directory.

The exit value of sort_extracts will be 0 if all files are transferred successfully. None 0 values may indicate the number of files which were believed to be duplicates and were not transferred.

The history of installed extracts must be checked carefully. Do not load duplicate files, omit extracts or load them in the wrong order.

Loading extracts with "loadd"

Once all the weekly extracts to be loaded are present in ./sorted and list correctly, the "loadd" command may be run.

     ./loadd&.
  

The script is normally run as a background process. The sorted llpg extracts are transferred from ./sorted to the ./src directory one at a time. The script "loadd" will wait until the Authority/Flare "ADL" process has loaded the extract into the database and removed the source file before proceeding to the next transfer.

The extracts should not be bulk loaded to "./src". At the time of writing Authority/Flare fails to load multiple files in the correct order. The use of "loadd" controls the uploads preventing the Authority/Flare error.

Running ADL

The Authority/Flare LPG import utility is called "ADL". ADL should be run with the settings should be as shewn below.

Screen dump of Authority/Flare ADL

Summary

	Save files to //flare/tool/llpg/extracts
	Start "ADL" on a flare client.
	Login in to the flare server and run the scripts to sort and load
        the extracts.

		ssh flare 
		sudo su - llpg
		./loadd&
		./sort_extracts	
   

C W Fulford
23rd August 2004. Updated 1st May 2006.


The layout and associated style sheets for this page are taken from the World Wide Web Consortium and used here under the W3C software licence.