Skip to content
/ TLE Public

Helper for data extract-transform-load (ETL)

License

Notifications You must be signed in to change notification settings

tebelorg/TLE

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

54 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

TLE

TLE is a helper for data extract-transform-load (ETL)

Sample ETL

Why This

TLE lets you programmatically transform spreadsheet data (or MySQL database) by remapping columns and preparing data (eg changing case-formatting) for consumption by downstream systems.

Originally developed pro bono for syncing ipcLink system used by Singapore non-profits to MYOB accounting system. For open-source ETL tools that are based on GUI, consider using Pentaho Data Integration or Talend Open Studio.

Set Up

Create a transformation mapping file with 2 rows, for eg

Name Email User ID
B LOWER(C) UPPER(F)
  • first row labelling the header fields desired in the output
  • second row specifying which column in input to grab data
  • use UPPER, LOWER, TITLE, SENTENCE for formatting case

For database input, provide input file in below format for script to read from database

TLE_COMMENTS DB_SERVER DB_USER DB_PASSWORD DB_NAME DB_TABLE
user comments servername username password database tablename

To Use

To transform input.csv using mapping.csv logic into output.csv (supported data formats - csv xls xlsx html)

php etl_start.php input.csv mapping.csv output.csv

To call TLE within a PHP script, simply assign the variables accordingly and include etl_start.php

$argv[1] = "input.csv"; $argv[2] = "mapping.csv"; $argv[3] = "output.csv"; include('etl_start.php');

License

TLE is open-source software released under the MIT license