ProjectTemplate can automatically load a variety of text based file formats, including comma separated value (CSV) files, tab separated value (TSV) files and generic whitespace separated value (WSV) files. In addition, automatic data loading is supported for several binary file formats, including the RData, SPSS, Stata and SAS formats.

Beyond those file formats, several ad hoc file types support the loading of data sets that are accessible over HTTP or contained in SQL databases, such as MySQL and sqlite.

Please note that several of file formats have not been tested yet, including Weka files, DBF files, EPIInfo files, MTP files, Octave files, Systat files and SAS files. Because ProjectTemplate is simply wrapping the ‘foreign’ package, these file formats are expected to work, but we have not confirmed that yet. Your mileage may vary.

Supported File Extensions

Ad Hoc File Types

URL Files

You can access CSV files over HTTP using the .url file extension. Inside of the .url file, you must place DCF that describes your data sources. An example file is shown below:

    url: http://www.johnmyleswhite.com/ProjectTemplate/sample_data.csv
SQL Files

ProjectTemplate supports access to many of the most common databases. All databases use the .sql file extension. Inside of the .sql file, you must place DCF that describes the connection protocol for your database. Example files for the support databases are shown below.

MySQL:
    type: mysql
    user: sample_user
    password: sample_password
    host: localhost
    dbname: sample_database
    table: sample_table
SQLite:
    type: sqlite
    dbname: /path/to/sample_database
    table: sample_table

    type: sqlite
    dbname: /path/to/sample_database
    query: SELECT * FROM users WHERE user_active == 1
PostgreSQL:
    type: postgres
    user: sample_user
    password: sample_password
    host: localhost
    dbname: sample_database
    table: sample_table
ODBC:
    type: odbc
    dsn: sample_dsn
    user: sample_user
    password: sample_password
    dbname: sample_database
    query: SELECT * FROM sample_table
Oracle:
    type: oracle
    user: sample_user
    password: sample_password
    dbname: sample_database
    table: sample_table
JDBC:
    type: jdbc
    class: org.jdbc.OracleDriver
    classpath: /path/to/ojdbc5.jar (or set in CLASSPATH)
    user: scott
    password: tiger
    url: jdbc:oracle:thin:@@myhost:1521:orcl
    query: SELECT * FROM emp
Heroku PostgreSQL:

This is a special case of the JDBC driver. It requires the current PostgreSQL JDBC jar file.

    type: heroku
    classpath: /path/to/jdbc4.jar (or set in CLASSPATH)
    user: scott
    password: tiger
    host: heroku.postgres.url
    port: 1234
    dbname: herokudb 
    query: select * from emp
.file Files

You can load data that is not stored in the current project using a .file file. You must specify the path and the extension that the file would have, if it were being loaded by the standard ProjectTemplate auto-loader. An example is shown below that would load an SQLite3 database stored in a separate location:

    path: /path/to/sample_database
    extension: db

Future Support For Data Sources

It is possible to provide support for new data sources by hooking into ProjectTemplate. The ElasticSearch reader is a working example of how to achieve this. We are looking forward to linking to your custom readers for new data sources, such as SQL Server, MongoDB or CouchDB. Please use the mailing list to get in touch with us.