It started with a simple goal: Put my interactive matplotlib desktop application on the web so that others could interact with it easily. I succeeded in the end but it was alot of work which pushed the boundaries of my technology knowledge and programming skill. Since it was alot of work, I'll organize the story of my journey into steps:
0. Try out different (existing) ways of putting matplotlib on the web.
I tried IPython errr Jupiter with different matplotlib HTML5 backends. While certainly cool, none of them implemented GUI functionality. That is, I would get an error because they did not respond to mouse clicks. I'm not starting with zero to be nerdy: Had I gotten IPython to work with mouse position clicks I would not have bothered with developing my solution explained in the following steps. This is also considering that IPython is a poor fit since it's a document while my program can be considered an 'app'.
1. Use a matplotlib backend that supports a GUI that can be displayed in a browser: GTK3
After some research, I settled on using GTK3 which comes with the broadway display server which can serve (individually) the application in an HTML5 page. After playing with GTK3 on Ubuntu Linux I managed to get my program to display in the browser as well as any other GTK3 program such as gEdit. It required a little change in matplotlib.
2. Serve the same program to any user who requests it
This was the most difficult part. The same application needed to be served on demand multiple times, perhaps simultaneously. Just pointing users to the display server is not sufficient. The (normal) overall process needs to be: 1. user requests application 2. display server with the application is started 3. monitor connection to disconnect and clean up after user exits.
I broke down the problem into 'modules' that represent each of these three activities though I worked on them in the order below:
- 2) Display manager: Working on the display manger came naturally after being able to just display GTK3 applications (step 1). It manages the starting up and shutting down of the display servers and the application that run on them. This code is pretty independent of the other modules although I put in functionality keeping in mind what the other modules need to do.
- 3) Connection manager: I used websockets to periodically send out a request for the user to confirm that he is active. This happens by simultaneously executing python on the server and javascript on the client. On disconnecting, the display and its associated application is stopped.
- 1) Request handler: It starts the process of starting a display when the user requests it.
Of course, that's easier said than done. The challenge in coding the display manger was managing the Linux processes and cleanly killing them. The connection manager had me go into event-driven programming and websockets. The request handler was rather straightforward to implement. But I had to extract the javascript from the broadway server to integrate it with my the process. So the webpage is (actually) served from the request handler and not the broadway server. I used the Tornado web framework to program the request handler and the connection manager. While the display manager is associated with the request handler (in the same program). Making sure events were coordinated was difficult!
3. Deploy
Having heard about about how awesome Docker is, I decided to recreate my development environment in a docker container. It was a bit tricky to get the networking working but I'm impressed with what is possible with Docker. Currently, my container is living in tutum.co while actually being served from AWS. Please volunteer to serve my application!
Reflections: Scientific Python on the Web
I have to say that I'm pretty satisfied with the result even though it's not ideal. As soon as as I got the task accomplished, I stopped working on it even though I could sink alot more time in it to improve quality, usability, and flexibility. Personally, I learned ALOT working on this project.
But I was disappointed that over in the R world you can create apps with GUIs on the web much more easily with shiny. They recently added mouse position clicks. I say it's disappointing because python is associated with multiple application domains including GUIs. Now in my research for solutions, I found ways to integrate matplotlib into GUIs but that would require some GUI expertise. There is no obvious solution for people used to the scientific python stack as to which GUI framework to use if web publishing is a concern. The people used to the scientific stack are not GUI experts nor are they web developers.
Having said that, what I like about my solution is that there is a path from matplotlib to a full GTK3 GUI application. So you can start with (simple) matplotlib elements and then if you decide you need the functionality of a real GUI you can integrate your work into the GTK3 framework. I've tried it. So you could have an app that runs on the desktop as well as the web. That is superior to shiny.
Some people have commented on the state of scientific python on the web. As part of the solution, I think somehow documents (html, IPython) and applications (GUIs) need to merge. The web has become a medium to deliver experiences.
Unfortunately, for delivering interactive python on the web, there is still alot of work to be done. But just by myself, I was able to deliver a product, albeit hacky, that serves python applications on the web using open source: Docker, Tornado, scipy/numpy, matplotlib, GTK3, Ubuntu, Linux...etc. Imagine what would happen if the open source community came together to work on this problem. Some components exist but now it has to come together. Hopefully, an open-source solution can be superior to a proprietary one.
---
Introduced at DC Python meetup.
No comments:
Post a Comment