Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Thursday, July 24, 2014

Compiling Hadoop from source sucks!

As I've discovered, Hadoop is not easy to setup let alone compile properly. For some reason the Apache Hadoop distribution doesn't include native libraries for 64-bit Linux. Furthermore, the included 32-bit native library does not include the Snappy compression algorithm. If Hadoop does not find these libraries in native form, it falls back to, I guess, slow or slower java implementations.

So, being the execution speed demon that I am, I went ahead and compiled Hadoop 2.4.1 from source on 64-bit Linux. It was a rough ride!

I generally followed this guide but preferring to download and compile Snappy from source, and yum installing java-1.7.0-openjdk-devel(1). I Used RHEL7 64-bit(2).

After getting the prerequisites, the magic command to compile is:
mvn clean install -Pdist,native,src -DskipTests -Dtar -Dmaven.javadoc.skip=true -Drequire.snappy -Dcompile.native=true (3)

You'll find the binaries in <your hadoop src>/hadoop-dist/target. I checked access to native libraries by issuing hadoop checknative after exporting the appropriate environment variables (such as in .bashrc. Refer to a Hadoop setup guide).


Non-obvious solutions to difficulties:
(1). yum install java-1.7.0-openjdk won't do! yea openjD(evelopment)k-devel makes sense!
(2). Gave up with Ubuntu
(3). -Dcompile.native=true was responsible for including the native calls to snappy in libhadoop.so. I did not see this in any guide on building Hadoop! Also, my compile process ran out of memory making javadocs, so I skipped it with -Dmaven.javadoc.skip=true

On a personal note, I really got frustrated with trying different things out but I had a sense of satisfaction in the end. It took me 4 days to figure out the issues and I know a thing or two about Linux!
xkcd

Tuesday, November 6, 2012

Making My Near-Field Radiation Calculator Application for nanohub.org

I wanted to share a bit of my experience developing my python application for nanohub.org. For me, there were two main issues: One is configuring the nanohub computing environment to resemble my development environment (and you'll need to figure out the details of installing your tool by looking at other tools and some docs that can help). The other is using Rappture to implement the functionality I want for my application.

To set up my development environment in nanohub, I first delved into how I could make a self-contained python installation. This had me looking into pythonbrew and virtualenv with pythonbrew being more useful. But the admins at nanohub were willing to install the packages that I needed so I didn't have to mess with that too much. Nonetheless, these two projects are great for creating portable python projects.

After I took care of the development environment, I went ahead and started learning about Rappture. I needed some flexibility and what I call 'dynamic input'. Now I could have made a simple application but then it wouldn't be as useful. I had to bend the API to suit my needs. I ended up merging what should have been two simple applications into one more complicated application. That is: I had two sets of inputs for the same calculation. I'll list the issues I had with Rappture:
  • Lacking documentation especially for the Python API. This is compensated for by looking at the API for the other languages and the many examples. I mainly figured out how Rappture works by example.
  • Lack of computation status output as addressed by this page.
  • Not all GUI elements are in the GUI GUI builder.
  • Could not generate an authentic contour plot. I used the elements field and cloud to generate a contour plot. This probably wasn't the intended use of the of the elements but it worked...horribly. It struggled to display just a 50x50 grid calculation not to mention that I couldn't put axis labels. Also of general note, elements take strings to input numbers which is inefficient (but works). In my case, I wanted to be able to display at least 1000x1000 points. My solution was to ditch this GUI element and just use an image generated by matplotlib's contourf.
  • 'Static' input. I wanted a user of my program to be able to change and check inputs in certain ways.
    • I wanted the user to be able to add a selection to a drop-down list. Could not be done. Instead, I have the user reviewing a table of named text input so that the name can go into a text box which should have been a selection.
    • Have an input value checked based on another input value. Can be done functionally by writing code that gets processed after hitting the 'Simulate' button. I don't know how I could hook into the GUI input elements so that they can be checked before the inputs are processed for simulation. For example, in my program I wanted to check that the value of 1.0 wasn't between two input values.
  • Limited selection of units. (eg. no rad/s.) There should be functionality so that any unit can be input and can be converted to a compatible unit.
Despite these concerns, I believe Rappture is great for those simple (from an input/output point of view) programs that were originally designed for textual input and output. For more complicated programs, Rappture poses limitations. But then there are other developed and more sophisticated GUI APIs out there. So a better Rappture would be duplicating efforts of those other APIs.

Here's how it works:
First you get a page where you can enter named electric permittivity functions.

Then you input the simulation parameters and options.
Then you get various outputs. I'm showing the cool ones here:
The characteristic nearly monochromatic near-field radiation between SiC and SiC
The above graph is an integration over the y-axis of the following function in two variables.



Friday, July 6, 2012

SFTP NetDrive

Scientific programs are usually targeted for linux systems. The problem is, I like working within Windows and I have to manage a linux system within a terminal window. I only execute in linux but I do everything else related to scientific work in Windows like creating graphs, writing up, and coding. Now, when I want to develop python code to run on the remote linux system, I have to have the same packages on my windows machine. Sometimes this isn't always possible (like python-meep because it's impractical (PITA) to build MEEP on anything but debian/ubuntu).

Now try this unique program to mount a Linux file system on your Windows machine over ssh.