User:DaveDixon/Dexy

Dexy is a multi-purpose project automation tool. Dexy is written in Python and has a command-line interface. Dexy is open source software with an MIT license.

Document automation
Dexy automates the repetitive parts of technical document creation. Target documents can be created with a text editor or a plain-text-compatible word processor such as LyX. Authors insert Dexy-compatible templating tags indicating the content to be included. The Dexy configuration file manages the process of creating document content such as syntax-highlighted source code and program output. At document-creation time, Dexy's dependency manager ensures that all code snippets, graphics, and output reflect the current state of the project. The configuration file indicates what commands to run to create the output, in what order, and with what parameters. Commands, known as filters, may make use of Python libraries or interact with external command line interfaces. Dexy supports collaborative development with platform independence, configuration file portability and compatibility with standard version control tools.

Document automation example
A code author is creating a blog post with examples showing how to use the API. Dexy will automatically:


 * run the example code, saving the results
 * apply syntax highlighting to the example code (using a pygments filter)
 * insert the results of API calls and the prettified example code into the blog post (using a jinja filter)
 * convert the markdown-formatted blog post to HTML (using a markdown filter or a pandoc filter).
 * upload the HTML to the WordPress API in draft mode (using a WordPress API filter).
 * publish the blog post when the author is satisfied with it.

Dexy architecture
The principle components of Dexy are the Dexy engine, the Dexy configuration file, and Dexy filters. The Dexy engine is written in Python and can be installed with either pip or easy_install. The dexy command starts the Dexy engine, which reads the configuration file and executes the filters and/or scripts in the specified order.

Dexy configuration file
The Dexy configuration file is a text or YAML-format file documenting dependencies, and includes calls to filters or scripts and the parameters to be passed to them. The execution order of filters and scripts is specified in the configuration file. Any developer with a copy of the component files and Dexy configuraiton file can build the target document with a simple command and the results will be consistent. Commands can be pipelined, and generally consist of calls to Dexy filters.

Dexy configuration file examples

 * index.md|markdown will apply the markdown filter to any files named index.md.
 * .java|java will apply the java filter to all files in the project with extension .java.
 * posts/2013/09/blog-post-about-dexy.html|wordpress will apply the wordpress filter to the file posts/2013/09/blog-post-about-dexy.html.

Commands may be Dexy filters or scripts, and can be pipelined, meaning they will be run in sequence, each taking the previous filter's output as its input.


 * posts/*.md|markdown|wordpress will first apply the markdown filter to convert the blog posts to HTML, then apply the wordpress filter to publish each blog post.

Dexy provides a variety of filters, and additional filters can be added using the Dexy plugin system. Filters can be written directly in Python, or in other languages using an executable script.

Dexy filters
Dexy uses filters to transform inputs to outputs. Under many circumstances, scripts can be used much like filters in Dexy.

Dexy filter examples

 * The java filter takes Java source code as input, and emits text as output (whatever was printed to STDOUT by running the Java code, if anything). This filter wraps the javac and java command line tools.
 * The markdown filter takes a markdown-formatted document as input, and emits HTML as output. This filter wraps the python markdown library.
 * The py filter runs python code and returns STDOUT.
 * The pycon filter runs Python code in the python REPL and returns a full transcript with prompts.
 * The r filter runs an R script.
 * The wordpress filter takes HTML as input, and emits information about the generated blog post as output (in JSON format). This filter wraps the WordPress API.

Dexy script example
A script can interact with other files in the project, such as opening files to read data and generating new files which then become part of the project. For example, to use R to make a graph of some data which is in a CSV file, the Dexy could:


 * create a script named example.R which opens data.csv and generates plot.png
 * specify the csv file as an input to (example.R|r in the Dexy configuration file)
 * run dexy

After running Dexy, plot.png will be included in the dexy output along with all the other generated files.