Tuesday, January 10, 2012

Parameter-study management with Python

A common task in scientific computing is running a set of scripts with different variable values each time. For example, say you have variables a and b and you want to create 4 sets of scripts so that
a=  b=
1    1
2    1
1   2
2   2

So the problem is:
1. generating the set of scripts quickly.
2. keeping track of folders
 - deleting folders corresponding to a certain set
 - view the correspondence of each folder to a variable set.
3. analyzing the output consistently

This is what motivated me to create "PSMT" (parameter study management tools). The code creates the example set by inputting a=[1,2] and b=[1,2]. Each set is in a separate numbered folder (no need to name the folders yourself or come up with a naming scheme!). There is an attribute (dict) for the object you create that stores the folder and the associated variable values

a=  b=  folder
1    1     \0

2    1     \1
1   2      \2
2   2      \3

Now you have your output organized! This gives structure to disparate files. Access the data in the analysis via either the folder number or by specifying the variables! The code separates the script mgt from the analysis since you're probably going to do these two processes on different computers.

This is the basic idea. I've documented my code pythonically but it's probably (lacking. I'm willing to answer questions. I've documented all the functions that the user is interested in, prefixed by user_ (eg. user_acoolfunction).
https://github.com/majidaldo/psmt
---
In the repository, I've also included a dictionary object that stores its items as separate files on the filesystem. The keys should be alphanumeric. The dictionary can be nested. I wanted a simple way to store hierarchies without using a database.

No comments:

Post a Comment