Thesis: Unsupervised Anomaly Detection in Sequences Using Long Short Term Memory Recurrent Neural Networks

http://hdl.handle.net/1920/10250

As a formalization of my previous post, I'm happy to publish my (second) Master's thesis.

Recurrent Neural Networks Can Detect Anomalies in Time Series

A recurrent neural network is trained on the blue line (which is some kind of physiologic signal). It has some kind of pattern to it except at t=~300 where it shows 'anomalous' behavior. The green line (not same scale) represents the error between the (original) signal and a reconstructed version of it from the neural network. At ~300, the network could not reconstruct the signal, so the error there becomes significantly higher.

Why is this cool??

• unsupervised: I did not care about data with anomalies vs data without anomalies
• trained with anomaly in the data: as long as most of the data is normal, the algorithm seemed robust enough to have learned the pattern of the data with the anomaly in it.
• no domain knowledge applied: no expert in this kind of time series provided input on how to analyze this data

More details for the more technical people:

- training algo: RMSprop

- input noise added

- the network is an LSTM autoencoder

- it's a fairly small network

- code: theanets 

And that's my master's thesis in one graph!


---
update 12/2018:
This post has been getting much attention which I appreciate. However, I find myself obligated to point readers to the latest research which obviates RNNs. Here is a great introduction towards the latest research The Fall of RNN LSTM.

Run CUDA applications on CoreOS

[https://github.com/majidaldo/coreos-nvidia]

Use this Dockerfile to install NVIDIA drivers and CUDA on more recent versions of CoreOS. It works by installing the NVIDIA Linux kernel module using plain Linux kernel source (containers see the kernel of the host OS, not the kernel of the container OS).

There are other Dockerfiles that manage this but they ask that you juggle two installations of the driver: one on the host and the other in the container. With the Dockerfile that I've developed, you only have one driver installation to worry about.

Commentary:

I find having to do this a bit hacky and against the containerization philosophy. Having the kernel module loaded from a Dockerfile and then, as a consequence, not being able to have multiple driver versions on the host. But maybe I'm asking too much from Docker's virtualization technique as I don't think it was meant to virtualize such low-level functions of the operating system.

Still, it's not that bad. Being able to use other CUDA-enabled Dockerfiles with only slight modification is great. I can also load and unload the kernel module at will. You just can't have two versions of the module running at the same time which isn't too much of an issue with GPU computing as you're probably going to not leave enough resources for other GPU processes on the (same) host.


Credits:
https://github.com/coreos/coreos-overlay/issues/924
http://tleyden.github.io/blog/2014/11/04/coreos-with-nvidia-cuda-gpu-drivers/

"Personal Compute Cloud" Infrastructure Code

[https://github.com/majidaldo/personal-compute-cloud]

Problem: Automate computing infrastructure setup
Solution: Docker hosts on CoreOS machines provisioned with Ansible.

I've recently finished coding up a solution to tackle 'personal' distributed computing. I was bothered by the (apparent) lack of a framework to handle the coordination of setting up multiple machines. And shell scripts are messy. Once I learned Ansible, I was not bothered! (It will be the only systems automation tool I will be using in the foreseeable future! yah..Ansible is AWESOME!)


Catering to the Scientific Computing Workflow: However, mere automation was not my only concern. I wanted a seamless transition from what I'm working on locally to being able to bring more computing power from remote machines. Unlike (pure) software engineering there isn't a 'development' environment and a 'production' environment. Now there are a handful of codes out there that can help you provision CoreOS clusters, but that does not fit well with the scientific computing workflow.

Status: Most of the functionality that I had planned has been implemented. However, like all codes, it's a work-in-progress. I'll be adding functionality as needed by my priorities.

Try it out.

Use Vagrant FROM Ansible to Automate Hybrid Cloud Infrastructure

[https://github.com/majidaldo/ansible-vagrant]


The Intro

This is NOT about having Vagrant provision with Ansible. This is about having Ansible treat Vagrant as a provider of hosts.

Building on my previous experience with the 'cloud', I still felt like I needed another tool to script and glue the process of getting my infrastructure up. I started out with shell scripts but they quickly got messy as the complexity increased. I knew about all the devops tools out there but I avoided them because I thought they would be too complex themselves for what I wanted to do which is relatively simple. But I bit the bullet on went full-on devops with Ansible.

Ansible is GREAT! I found it suitable for (technically-minded) beginners. However, it still took me a few days to get the hang of it. I had to get a little bit under the hood since it did not do what I wanted it to do out of the box.

I want to setup something like a hybrid cloud where I run some services locally and just bring up high-performance compute nodes on demand and have them talk with my local services. I use Vagrant to setup local virtual machines. Vagrant is great for development environments but when I want to manage and orchestrate several VMs locally (let alone on the cloud), things can get messy.

So, I (further) developed ansible-vagrant to interface with Vagrant from Ansible (solving cygwin problems along the way).


The Cream

You can, from Ansible

• Set state=(up|halt) for some VM
• Get a Vagrant host inventory
• Get a SSH config for a host
• Destroy VMs

CoreOS cloud-config Generator

[https://github.com/majidaldo/cloudconfig-writer]

After moving my workflow over to Docker I realized that it's not a complete solution. There is still the process that comes before Docker that must be addressed: namely, provisioning and managing machines on a (ugghh) "cloud" provider such as Amazon EC2, Digital Ocean, and even Vagrant. Also, CoreOS has become an important player in the Docker scene, providing just the minimum operating system needed to run Docker in a cluster environment. CoreOS also provides some commonality in the process of provisioning machines on different providers by making use of a cloud-config file; The same (or almost the same) cloud-config file can be used on different providers.

That's good news but it introduces problems like:
- I want to keep this piece of the cloud-config file out of source control
- I don't want to copy/paste between cloud-config files (DRY issues) :
    - this cloud-config file is just like this one but with a different password/user/hostname/IP address
    - this cloud-config file is just like this one but with an added section

So, I made a program to address these issues! Find out more by reading the README in the repository.

Comments are welcome.

Serving GTK3 Applications on the Web

....Scientific Python on the Web: not there yet!

It started with a simple goal: Put my interactive matplotlib desktop application on the web so that others could interact with it easily. I succeeded in the end but it was alot of work which pushed the boundaries of my technology knowledge and programming skill. Since it was alot of work, I'll organize the story of my journey into steps:


0. Try out different (existing) ways of putting matplotlib on the web.

I tried IPython errr Jupiter with different matplotlib HTML5 backends. While certainly cool, none of them implemented GUI functionality. That is, I would get an error because they did not respond to mouse clicks. I'm not starting with zero to be nerdy: Had I gotten IPython to work with mouse position clicks I would not have bothered with developing my solution explained in the following steps. This is also considering that IPython is a poor fit since it's a document while my program can be considered an 'app'.


1. Use a matplotlib backend that supports a GUI that can be displayed in a browser: GTK3

After some research, I settled on using GTK3 which comes with the broadway display server which can serve (individually) the application in an HTML5 page. After playing with GTK3 on Ubuntu Linux I managed to get my program to display in the browser as well as any other GTK3 program such as gEdit. It required a little change in matplotlib.


2. Serve the same program to any user who requests it

This was the most difficult part. The same application needed to be served on demand multiple times, perhaps simultaneously. Just pointing users to the display server is not sufficient. The (normal) overall process needs to be: 1. user requests application 2. display server with the application is started 3. monitor connection to disconnect and clean up after user exits.

I broke down the problem into 'modules' that represent each of these three activities though I worked on them in the order below:

• 2) Display manager: Working on the display manger came naturally after being able to just display GTK3 applications (step 1). It manages the starting up and shutting down of the display servers and the application that run on them. This code is pretty independent of the other modules although I put in functionality keeping in mind what the other modules need to do.
• 3) Connection manager: I used websockets to periodically send out a request for the user to confirm that he is active. This happens by simultaneously executing python on the server and javascript on the client. On disconnecting, the display and its associated application is stopped.
• 1) Request handler: It starts the process of starting a display when the user requests it.

Of course, that's easier said than done. The challenge in coding the display manger was managing the Linux processes and cleanly killing them. The connection manager had me go into event-driven programming and websockets. The request handler was rather straightforward to implement. But I had to extract the javascript from the broadway server to integrate it with my the process. So the webpage is (actually) served from the request handler and not the broadway server. I used the Tornado web framework to program the request handler and the connection manager. While the display manager is associated with the request handler (in the same program). Making sure events were coordinated was difficult!


3. Deploy

Having heard about about how awesome Docker is, I decided to recreate my development environment in a docker container. It was a bit tricky to get the networking working but I'm impressed with what is possible with Docker. Currently, my container is living in tutum.co while actually being served from AWS. Please volunteer to serve my application!


Reflections: Scientific Python on the Web

I have to say that I'm pretty satisfied with the result even though it's not ideal. As soon as as I got the task accomplished, I stopped working on it even though I could sink alot more time in it to improve quality, usability, and flexibility. Personally, I learned ALOT working on this project.

But I was disappointed that over in the R world you can create apps with GUIs on the web much more easily with shiny. They recently added mouse position clicks. I say it's disappointing because python is associated with multiple application domains including GUIs. Now in my research for solutions, I found ways to integrate matplotlib into GUIs but that would require some GUI expertise. There is no obvious solution for people used to the scientific python stack as to which GUI framework to use if web publishing is a concern. The people used to the scientific stack are not GUI experts nor are they web developers.

Having said that, what I like about my solution is that there is a path from matplotlib to a full GTK3 GUI application. So you can start with (simple) matplotlib elements and then if you decide you need the functionality of a real GUI you can integrate your work into the GTK3 framework. I've tried it. So you could have an app that runs on the desktop as well as the web. That is superior to shiny.

Some people have commented on the state of scientific python on the web. As part of the solution, I think somehow documents (html, IPython) and applications (GUIs) need to merge. The web has become a medium to deliver experiences.

Unfortunately, for delivering interactive python on the web, there is still alot of work to be done. But just by myself, I was able to deliver a product, albeit hacky, that serves python applications on the web using open source: Docker, Tornado, scipy/numpy, matplotlib, GTK3, Ubuntu, Linux...etc. Imagine what would happen if the open source community came together to work on this problem. Some components exist but now it has to come together. Hopefully, an open-source solution can be superior to a proprietary one.


---
Introduced at DC Python meetup.

Bayesian Optimization Demo Game: Can you beat the 'computer'?

...an interactive matplotlib (GTK3) app served on the web.
app (not working properly on Chrome): [http://bayeisan-optimization.25800bfd.svc.dockerapp.io:8000/bo]
[https://github.com/majidaldo/GTK3webserver]

I recently made a notebook about Gaussian processes (GP). Now, Gaussian processes are used by Bayesian optimization. In Bayesian optimization, the goal is to optimize a function with as few (probably expensive) function evaluations as possible. Here's a good tutorial.

To see how it works, I implemented a basic optimizer following the mathematics introduced in the referenced tutorial (which wasn't as difficult as the math behind the GP!). I began from the code for the GP, to see how a Bayesian optimizer (BO) would optimize a toy 1-D problem.

But then, I wondered how the performance of a human would compare to the BO. So I made a game! In this game, the goal is to, of course, find the maximum of the function is as few tries as possible. Both players in each trial will attempt to find the maximum. A running average of the number of tries is kept.

After some experimentation playing the game, it seems there is a lower bound for the average number of tries needed. For the way I have my game set up (50 possible choices), the BO needed 10 point something +/-  point something tries. Furthermore, trying different so-called acquisition functions for the BO did not have much of an effect. This figure is based on thousands of trials (central limit theorem at work). And playing as the creator of my game, it was difficult for me to keep below 10 tries as an average. Of course, I can't play thousands of times.

These results imply that a human would not be able to optimize a function in three dimensions or more in less tries than a BO. There are simply too many variables to keep track of and it is very difficult if not impossible to visualize which is part of the point of using BOs.


Some features of the program:

• For aesthetic reasons, the y-axis has to be fixed based on the range of values of the function. But then the visual clue of having a point close to the top edge of the plot gives the human player an unfair advantage. The solution is to add some random margin between the maximum and the edge. So, if you happen to get a point that is close to the top edge, choose its neighbors!
• To generate a 'random' smooth function, I get some normally distributed points and sprinkle them on the domain. Then I use splines to connect them. It works surprisingly well!

Implementation notes:

The programming started in a functional style which is what you'd expect out of mathematical code and it's what I'm used to. However, once I got into matplotlib's GUI stuff, things started to get a little messy as a GUI requires reactive event-driven programming. Both styles of programming exist in my program and they intersect at while loops. You can run the game script with python on your desktop from the command line.

Now, the work involved in putting the game on the web, so that you, the reader, can easily engage in it, deserves its own post or two! Once I had the 'desktop' application running, I swam through oceans of technology until I got it to the form presented here. I should mention here that I came across a program similar to mine on Wolfram|Alpha but I can't find it anymore.

So here is the game served on the web. It's a Docker container managed for free, for now, by tutum.co but hosted almost for free on AWS for a few more months as of the date of this post. If you can spare 1GB of HD space and 100MB of memory to host my program, let me know!

C vs. Python in the Context of Reading a Simple Text Data File

Problem: Using C, read a Matrix Market file generically and versatilely.
Generically: using the same code (at run-time), interpret a line of data in various ways

I'm trying to do this in C. Coming from Python, C makes me feel CRIPPLED. Laundry list of offenses:

• Primitive string operations...oh actually they are just character arrays.
  example: char teststring[10]="asdf"; (OK initialization) teststring="qwer"; (FAIL what I expect is a reassignment fails..turns out you must call string copy and make sure the destination has enough space.).
• No negative indexing
• Can't get length of an array
• Can't handle types as objects. You can't say: if this object is of this type, then do this. Types are 'hardcoded'. You can't return a type.
• Case labels must be known at compile-time. The C compiler can't understand that if I use a function output with a known input at compile time to make a label that it could be known at compile-time. As a mathematically-inclined programmer, all I care about is functionality. I don't like to think about the preprocessor as a separate process.
• Your program can compile and work but in the realm of "undefined behavior". Good luck finding a bug caused by "undefined behavior".

Which brings me to some C items that have overlapping functionality: enum, arrays, and X-Macros. They can all be viewed as lists, but macros live in the preprocessor; you cannot iterate over enums and they are essentially only a list of integers; and if you want to know the size of an array, you can only do it at compile-time with a macro hack.  So if you need functionality on the same list that had to be implemented in the preprocessor and at run-time, you have to violate DRY.

While I do think lower-level functions are essential for some purposes, this exercise makes me appreciate Python more (yes particularly Python and not other interpreted languages). In Python, I can read the lines in as a matrix of strings, then I could vectorize type conversion operations in a few lines of code.

So, I've resorted to code generation to create functions for all cases which solves half the problem. I won't bother trying to make things more generic. If it could be done, I expect that it's going to look messy.

4/17 Update: I've finished my reader. The "main()" file is here. I think I organized my procedure well but it took alot of code. It's ridiculous. I could accomplish the same with acceptable speed in about two dozen lines of Python code. I know this because I've written code to read data generically in Python several times.

Making My Near-Field Radiation Calculator Application for nanohub.org

I wanted to share a bit of my experience developing my python application for nanohub.org. For me, there were two main issues: One is configuring the nanohub computing environment to resemble my development environment (and you'll need to figure out the details of installing your tool by looking at other tools and some docs that can help). The other is using Rappture to implement the functionality I want for my application.

To set up my development environment in nanohub, I first delved into how I could make a self-contained python installation. This had me looking into pythonbrew and virtualenv with pythonbrew being more useful. But the admins at nanohub were willing to install the packages that I needed so I didn't have to mess with that too much. Nonetheless, these two projects are great for creating portable python projects.

After I took care of the development environment, I went ahead and started learning about Rappture. I needed some flexibility and what I call 'dynamic input'. Now I could have made a simple application but then it wouldn't be as useful. I had to bend the API to suit my needs. I ended up merging what should have been two simple applications into one more complicated application. That is: I had two sets of inputs for the same calculation. I'll list the issues I had with Rappture:

• Lacking documentation especially for the Python API. This is compensated for by looking at the API for the other languages and the many examples. I mainly figured out how Rappture works by example.
• Lack of computation status output as addressed by this page.
• Not all GUI elements are in the GUI GUI builder.
• Could not generate an authentic contour plot. I used the elements field and cloud to generate a contour plot. This probably wasn't the intended use of the of the elements but it worked...horribly. It struggled to display just a 50x50 grid calculation not to mention that I couldn't put axis labels. Also of general note, elements take strings to input numbers which is inefficient (but works). In my case, I wanted to be able to display at least 1000x1000 points. My solution was to ditch this GUI element and just use an image generated by matplotlib's contourf.
• 'Static' input. I wanted a user of my program to be able to change and check inputs in certain ways.
• I wanted the user to be able to add a selection to a drop-down list. Could not be done. Instead, I have the user reviewing a table of named text input so that the name can go into a text box which should have been a selection.
• Have an input value checked based on another input value. Can be done functionally by writing code that gets processed after hitting the 'Simulate' button. I don't know how I could hook into the GUI input elements so that they can be checked before the inputs are processed for simulation. For example, in my program I wanted to check that the value of 1.0 wasn't between two input values.
• Limited selection of units. (eg. no rad/s.) There should be functionality so that any unit can be input and can be converted to a compatible unit.

Despite these concerns, I believe Rappture is great for those simple (from an input/output point of view) programs that were originally designed for textual input and output. For more complicated programs, Rappture poses limitations. But then there are other developed and more sophisticated GUI APIs out there. So a better Rappture would be duplicating efforts of those other APIs.

Here's how it works:
First you get a page where you can enter named electric permittivity functions.

Then you input the simulation parameters and options.

Then you get various outputs. I'm showing the cool ones here:

The characteristic nearly monochromatic near-field radiation between SiC and SiC

The above graph is an integration over the y-axis of the following function in two variables.

PBS management tools

https://github.com/majidaldo/pbsmgr
This is related to my parameter study mgt. code.  The problem is how to keep track of and resubmit interrupted jobs.

The usage scenario I'll describe here illustrates the motivation of the code:

Create the task array (eg. folders 0-3). Each task has a pbs script with "pbs_name 0/1/2/3" in the script to identify it in the pbs system. (You can create these unique names yourself or have them automatically generated by PSMT.)

cd into the parent of the folders. submit all pbs scripts in the subdirectories with

python pbsmgr.py

This will keep running and resubmit any jobs that are interrupted. So you should have your script rename or delete the pbs script when it's (actually) finished so that it doesn't get picked up my pbsmgr again.

You can watch the status of jobs with

python pbsmon.py

from the same directory. At first it will list all jobs but then it will only print /changes/ to job status.

LAMMPS tools: ave/time reader

Self explanatory.

vecfilesrdr in avetimevecrdr.py in https://github.com/majidaldo/lammpstools

LAMMPS tools: dumpfiles to hdf5 database

If you're going to manipulate large dump file data more than once, it's best to have it in the binary but portable hdf5 format. Requires h5py. Can resume if the process is interrupted.

dumps2hdf5 in dumps2hdf5.py in https://github.com/majidaldo/lammpstools. Can also be run from the command line. See code documentation

LAMMPS tools: dumpfile reader

function dumps2vecs in dumps2vecs.py in https://github.com/majidaldo/lammpstools

 I tried to make this reader as fast as possible since dump files can get huge. It can read one dump file or a sequence of them.

Parameter-study management with Python

A common task in scientific computing is running a set of scripts with different variable values each time. For example, say you have variables a and b and you want to create 4 sets of scripts so that
a=  b=
1    1
2    1
1   2
2   2

So the problem is:
1. generating the set of scripts quickly.
2. keeping track of folders
 - deleting folders corresponding to a certain set
 - view the correspondence of each folder to a variable set.
3. analyzing the output consistently

This is what motivated me to create "PSMT" (parameter study management tools). The code creates the example set by inputting a=[1,2] and b=[1,2]. Each set is in a separate numbered folder (no need to name the folders yourself or come up with a naming scheme!). There is an attribute (dict) for the object you create that stores the folder and the associated variable values

a=  b=  folder
1    1     \0

2    1     \1
1   2      \2
2   2      \3

Now you have your output organized! This gives structure to disparate files. Access the data in the analysis via either the folder number or by specifying the variables! The code separates the script mgt from the analysis since you're probably going to do these two processes on different computers.

This is the basic idea. I've documented my code pythonically but it's probably (lacking. I'm willing to answer questions. I've documented all the functions that the user is interested in, prefixed by user_ (eg. user_acoolfunction).
https://github.com/majidaldo/psmt
---
In the repository, I've also included a dictionary object that stores its items as separate files on the filesystem. The keys should be alphanumeric. The dictionary can be nested. I wanted a simple way to store hierarchies without using a database.