User:Bsambathkumar/sandbox

== A.Data Flow ==



1.	CSV Export

•	Stop data exported to DIGI by using CSV format.

•	FTP SERVER details can be changed on following Settings

public static $CSVStopData = array( 'tempDirPath'       => '/tmp/export/',  'csvFile'           => 'digiroad_stops.csv',  'timestampFile'     => 'flag.txt',  'ftpServer'         => '80.248.161.126',  'ftpUserName'       => 'karttakeskus',  'ftpPassword'       => 'encelent',  'ftpZipFilePath'    => 'koontikartta/test/output/',  'ftpZipFile'        => 'digiroad_stops.zip'							 ); tempDirPath - zip file extracted on this dir for processing

csvFile - CSV file name

timestampFile – Time stamp of CSV file creation (present time)

ftpServer, ftpUserName,  ftpPassword, ftpZipFilePath, ftpZipFile -  FTP server & file details

•	  Stops are taken in batch from DB. To control the number of stops exported on each batch change the following constant. const EXPORT_BATCH_DBROWS_COUNT = 500; •	  CSV data separated by following delimiter

const CSV_DELIMITER = ';'; •	All dataload activities logged on write/export_log_ .log file

•	CSV header will be taken from setting $CSVHeadersExport

2.	CSV Import

•	Stops details imported from DIGI by CSV format.

•	FT P SERVER details can be changed on following Settings

public static $DataLoadFileDetails = array( 'tempDirPath'       => '/tmp/import/',  'csvFile'           => 'digiroad_stops.csv',  'timestampFile'     => 'flag.txt',  'ftpServer'         => '80.248.161.126',  'ftpUserName'       => 'karttakeskus',  'ftpPassword'       => 'encelent',  'ftpZipFilePath'    => 'koontikartta/stops/',  'ftpZipFile'        => 'digiroad_stops.zip' );

tempDirPath - zip file extracted on this dir for processing

csvFile - CSV file

timestampFile – Time stamp of CSV file creation

ftpServer, ftpUserName,  ftpPassword, ftpZipFilePath, ftpZipFile -  FTP server & file s

•	Before fetching CSV file, timestamp file (flat.txt) has fetched to check the data validity. The timestamp has tested against the latest loadDataTime value from database.

•	CSV file headers should be listed on same order and spelling in the following settings $CSVHeadersImport

•	Delimiter of the CSV data

const CSV_DELIMITER = ';'; •	Each stop presented on each row in the CSV file. Import process loads the stops as batch to avoid higher memory usage. To control the number of stops on each batch process change the following settings.

const IMPORT_BATCH_CSVROWS_COUNT = 500;

•	All dataload activities logged on write/import_log_ .log file •	Stop data compared against the DB stop data.

If the stop not exist in the stop table, stop will be inserted.

If the stop exist in the stop table, DB stop data will be compared with CSV stop data.

Changes on fields other than priority fields will be consider for update and stop data updated.

•	Stop editor has more priority for stop coordinate and bearing. Stop editor primarily used for changing location and bearing. On DIGI, changes on other fields carried.

Stop data & status will be updated based on following condition,

1.	If the CSV fields are similar to DB stop field and status is ‘Edited’, the status will be changed to “Updated”.

2.	If the stop field status id ‘New’ or ‘Updated’, changes will considered for import.

3.	Stop name will be updated, if it has some value (not empty) from CSV.

4.	If the stop not exist in CSV and exist in DB, then this stop considered as expired. This stop valid_to date will be changed to previous day.

3.	XML Import

•	Received xml validated against the schema.

•	Import action will be logged on write/xml_import_log_ .log file

•	XML stop data parsed and compared against DB stop data.

•	If the stop not exists, inserts the stop.

•	If the stop exist, fields are compared and changed on priority fields are not considered. Priority fields can be modified on following setting, $PRIORITY_FIELDS

4.	Stop Edit update – XML Export

•	Update triggered on changing stop data on following conditions,

i.	Adding new stop

ii. Editing stop data

•	XML generated for the given stops based on schema

•	This activity logged on write/xml_export_log_ .log file

•	XML send to message server. Message server & xml schema can be set following setting,

public static $XMLStopData = array( 'schemaPath'                    => '',  'schemaFile'                    => 'StopSchema.xsd',  'xmlNameSpaceUrl'               => 'http://www.w3.org/2001/XMLSchema-instance',  'xmlNameSpaceLocationAttribute' => 'noNamespaceSchemaLocation',  'xmlMessageServerUrl'           => 'http://10.13.172.183/xml_stop/receiveXml.php'			   );

5.	Stop Edit update - Broadcast to VALLU / Stop Sequence system

6.	Stop Edit Fetch

While creating stop changes on stop editor, it should be intimated immediately to VALLU Ext users. This functionality has achieved by the following method,



Figure 1: Stop Change Auto Update

1.	VALLU Ext client receives the stop details on initial page load.

2.	Immediately it also creates HTTP long Polling AJAX request to Vallu server. This request runs the script (stopUpdate.php) in the Vallu server which checks for the stop changes with Memcache server.

3.	While producing any change with stops on stop editor, it pushes that stop details to MemCacheServer.

4.	stopUpdate.php Script takes the change from MemCacheServer and returns to VALLU EXT. This will be intimated to user and rerouting will be carried if required.

7.	Removing Stop Editor changes

JIRA ticket Reference - JPKKARTTA-446, JPKKARTTA-429

Problem: If the priority fields (bearing, xyCoordinate, bearingDescription) changed by using KSE, it cannot be overwritten by DIGI data during data load.

Client wants to remove the changes done by using KSE. So, that the data load can load the DIGI data to the database.

Solution:

Change history tab lists all the changes done on KSE. User can select the stop changes and can remove all the changes carried on KSE. User can remove all the changes done for stop (> 500000). After removal, the stop status will be ‘updated’. So this stop will be treated like DIGI stop and will receive update on all fields.

8.	Creating Stop with new Stop Id

JIRA ticket Reference - JPKKARTTA-446

User can edit the stop id for the stops with stop id < 500000.Text box provided to get new stop id on stop details page. If it is already exist, show the error. Otherwise, create the new stop with new stop id and old stop data. Link to old stop id added on new stop details page. Old stop id will be changed to deprecated status. This will be shown as read only mode. No more edits allowed. This will be avoided on CSV export. Link to new stop id will be given on stop details page. New stop id data with old stop id will be broadcasted as XML message to timetable editor and vallu. Timetable editor takes corrections on DB (find and replace all old stop id by new stop id)

Steps explained with following 2 cases,

Case 1: New Stop Id already exist.

i.) Old stop (validFrom & validTo) set with expired date.

ii.) Old stop will become read only (and has link to NewStopId) (new entry on table 'stopRelation')

iii) Send xml (old stop data with expired dates) to timetable editor with  tag.

iv.) Send xml(old stop data with expired dates) to vallu without  tag.

Case 2: New Stop Id not exist

i.) Old stop (validFrom & validTo) set with expired date.

ii.) New stop created with old stop data & with new stop Id

iii.) Old stop will become read only (and has link to NewStopId) (new entry on table 'stopRelation')

iv) Send xml (old stop data with expired dates) to timetable editor with  tag.

v.) Send xml(old stop data with expired dates) to vallu without  tag.

Timetable editor xml receiver

i.) Receives the xml and checks for 

ii.) If the  not exist, creates new stop id (with old stop data).

iii.) Replaces all the oldStopId occurrences by NewStopId on 'lineToStop' & 'departureToStop' tables.

iv.) Old stop (validFrom & validTo) set with expired date.

== B.Log Files ==

1.	DB Error log

•	All the db relevant error will be logged on write/debug.log file.

•	This can be set on following setting,

DEBUG_LOG_FILE = "/var/www/koontikartta_stop_editor/write/debug.log"; •	To enable the log, following setting should be set,

DEBUG_LOG = 1; 2.	Dataload Logs

•	CSV export logged on write/export_log_ .log file

•	CSV import logged on write/import_log_ .log file

•	XML Import logged on write/xml_import_log_ .log file

•	XML export logged on write/xml_export_log_ .log file