Thursday, February 06, 2014

Plotting hardware stats via plotly

I recently signed up to try the beta of Plotly (@plotlygraphs) and decided to try it out by plotting the temperature, CPU load and RAM usage of a Raspberry Pi. Temperature, particularly, was of interest to me as I intend to stream video from the camera module, which makes heavy use of both the GPU and CPU. Plotly provide a Python API, which suits me perfectly.

Note that I use Python 2.7 and this code requires some modification to work correctly in Python 3.

Step 1: Getting the stats

I wasn't intending to reinvent the wheel as far as getting the hardware stats goes, so I shamelessly stole code from here and here, as below.
from subprocess import PIPE, Popen
import psutil

def get_cpu_temp():
    tempFile = open("/sys/class/thermal/thermal_zone0/temp")
    cpu_temp = tempFile.read()
    tempFile.close()
    return cpu_temp / 1000.

def get_gpu_temp():
    process = Popen(['vcgencmd', 'measure_temp'], stdout=PIPE)
    output, _error = process.communicate()
    return float(output[output.index('=') + 1:output.rindex("'")])

def get_ram_used():
    return psutil.phymem_usage().percent

def get_cpu_usage():
    return psutil.cpu_percent()
These functions do exactly what you'd expect, so I won't go into any detail.

Step 2: Using the plotly API

To send the data to plotly, you first create a plotly object that provides a wrapper around their API, and then use methods of that object to add data or set the graph layout. The plotly object is created as follows:
USERNAME = 'username'
APIKEY   = 'api_key'

py = plotly.plotly(username_or_email=USERNAME, key=APIKEY)
You can find your API key by logging into plotly, clicking your username in the top right and clicking 'Settings'. Alternatively, visit the Python API page and expand the installation instructions.

The two important methods in the plotly object are layout and plot. My script creates a layout specification and sends it to plotly as follows.
layout = {
    'title'       : 'Camera 1 hardware',
    'xaxis'       : {'title':'Time', 'type':'date'},
    'yaxis'       : {'title':'Temperature (deg. C)'},
    'yaxis2'      : {'title':'% usage', 'side':'right', 'overlaying':'y'},
    'showlegend'  : True,
    'legend'      : {'x' : 1, 'y' : 1},
}

py.layout(layout, filename=filename)
The x and y values for the legend are in the range 0 to 1, with (0,0) being the bottom-left corner of the graph.

I also insert annotations at appropriate points using lines similar to the following (after which layout must be called again):
layout['annotations'] = [{
    'text'      : 'Camera enabled',
    'xref'      : 'x',
    'yref'      : 'y2',
    'x'         : calendar.timegm(curr_time.utctimetuple())*1000,
    'y'         : _cpu_load,
    'ax'        : -100,
    'ay'        : 10,
    'showarrow' : True,
    'arrowhead' : 7
}]
The ax and ay values control either the offset of the label or the length of the arrow, I'm not 100% clear on which but I would assume the latter, and they seem to be specified in pixels rather than axis units. Note that I convert my datetime object into a timestamp in milliseconds. While the plotly API supports datetime objects (and strings of the form 'YYYY-MM-DD HH:MM:SS') for datapoints, it currently doesn't for annotations. I've been told that this is on their todo list though.

I send actual datapoints to plotly using the following (reformatted somewhat to fit on this page).
cpu_temp = {'x': [curr_time], 'y': [get_cpu_temp()],
            'type': 'scatter', 'mode': 'lines',
            'name':'CPU temperature'}
gpu_temp = {'x': [curr_time], 'y': [get_gpu_temp()],
            'type': 'scatter', 'mode': 'lines',
            'name':'GPU temperature'}
ram_used = {'x': [curr_time], 'y': [get_ram_used()],
            'type': 'scatter', 'mode': 'lines',
            'name':'RAM usage', 'yaxis':'y2'}
cpu_load = {'x': [curr_time], 'y': [get_cpu_usage()],
            'type': 'scatter', 'mode': 'lines',
            'name':'CPU usage', 'yaxis':'y2'}

py.plot([cpu_temp, gpu_temp, ram_used, cpu_load],
        filename=filename, fileopt='extend')
There are two methods of passing data to plotly, one is to use individual lists of values such as
x1 = [1, 2, 3]
x2 = [1, 3, 5]
y1 = [1,3,6]
y2 = [6,3,1]
py.plot(x1, y1, x2, y2)
and the other is to use a list of dictionaries (one dictionary per trace) as I've done in my code. I use fileopt='extend' as I send the data for each trace one point at a time as it is gathered. The alternative would be to collect and buffer all the values and then send them all at once.

Putting all this together, along with some additional code to sample the values at the correct times, results in a graph looking like the following.



My blog doesn't give the graph much space to play with, but you can also view it directly on the plotly website, where it looks significantly better. For those interested, the graph was embedded using:
<iframe height="510" id="igraph" scrolling="no" seamless="seamless"
src="https://plot.ly/~Kemp/17/600/500/" width="610"></iframe>
The width and height to display at is controlled directly in the URL and I've added an extra 10 pixels to the iframe size to prevent scrollbars from appearing. When I was looking at the temperature under load I, of course, graphed the Pi over a much longer period of time. I performed this shorter version of the test in order to be able to show the graph reasonably well in a small space.

You can find the complete source for my script here. This was a quick investigation to satisfy my curiosity about the hardware, so it's not the most elegant script.

That's all there is to it. Once you have the basics down, the plotly API seems very easy to use. You can do a lot more than what I've covered here, such as graphing data from uploaded files, processing the data directly on the website, and generating many more types of graph than just scatter/line plots. If you need to create graphs to share with other people I'd suggest giving them a try.

Monday, December 16, 2013

Modelica socket communication (Part 1: socket library)

Now that I've demonstrated a method of using the OpenModelica CORBA interface to set up and run simulations, I will move on to a way of passing data out of and in to a running simulation (for example, to use a control algorithm that is more easily implemented in another language). I will use sockets for this as they are one of the simplest and most generic communication methods available.

The first step in setting up socket communication within a Modelica simulation is to create a socket library that can be called from the simulation code. Fortunately, this is very simple C code. You can find the relevant code here.

Now that we have the source file, we will compile the library. Instead of running the commands by hand, we will create a Makefile to do it for us. It's a little bit of extra effort for a huge saving later on if you come back to change things or recompile on a different machine. The makefile is a very simple one with the following content:

.PHONY: all


all: libomsocket.a


Socket.lo: Socket.c
 libtool --mode=compile gcc -c $<

libomsocket.a: Socket.lo
 libtool --mode=link gcc -o $@ $<

clean:
 -rm Socket.lo Socket.o
 -rm -rf .libs

Note that the clean target doesn't delete the final library. It is used for cleaning up the other miscellaneous files created during compilation and linking.

I used libtool to compile the library as it prevents me having the remember the full commands (I do this so little that I have to look them up). If you don't already have libtool then you should install it via your package manager (e.g. sudo apt-get install libtool in Ubuntu/Mint).

The socket code I provided is based on code found here. If you're interesting in the example given there then note that one of the links in the relevant post is not formatted correctly and one is missing. The complete set of files referenced in the post are:

OpenModelica and Python via CORBA (Part 3: Python wrapper)

Part 1 (getting started) | Part 2 (IDL compilation) | Part 3 (Python wrapper)

The final step in the process is to create a wrapper to allow easy programmatic control of OMC. I have posted a simple wrapper that provides the facilities required by the bouncing ball example here. It is not at the same level as the work by Ganeson et al. of course.

Using this wrapper, the bouncing ball example is much simpler than before, requiring only

import sys

from omc import OMC

omc = OMC()
omc.connect()

omc.clear()
print omc._send_command("model test end test;")
print omc.list_classes()
print omc.load_file("/usr/share/doc/omc/testmodels/BouncingBall.mo")
print omc.simulate("BouncingBall")
print omc.plot("h")

Note that my wrapper doesn't provide a specific method for directly injecting things like models. If it did, it would be a redirect to _send_command and so I've simply used that for this example.

With this fairly short third part, I will end the series here. Hopefully it provides a useful starting point for more complex projects.

Tuesday, November 26, 2013

Compiling OpenModelica on Mint

When compiling OpenModelica from source on Mint (and possibly on Ubuntu?) the compile cheat sheet in the readme gives working instructions with one exception: several packages, such as libqtwebkit-dev, aren't installed when you run the sudo apt-get build-dep openmodelica step. I had to use the following commands from the text further down the readme to get everything needed:

sudo apt-get install antlr libantlr-dev
sudo apt-get install libreadline-dev libqt4-dev libqtwebkit-dev libqwt5-qt4-dev
sudo apt-get install sqlite3 libsqlite3-dev


Note that I had to use to libreadline-dev instead of libreadline5-dev. Either way, that particular package was already installed, but I've left it here for completeness as the others on the same line weren't.

I also needed to install bison and flex for the testsuite (not mentioned in the build instructions).

There may be other packages required that I already had installed, so my advice would be to install all of the packages they suggest rather than relying on build-dep.

Thursday, October 31, 2013

OpenModelica and Python via CORBA (Part 2: IDL compilation)

Part 1 (getting started) | Part 2 (IDL compilation) | Part 3 (Python wrapper)

In part 1 I showed you how to get set up with the omniORB library/tools to communicate with OMC from Python. This post will show how to compile an IDL file to provide the appropriate interface to your script. This post assumes that the environment variables from last time are still set up and that you have OMC running with the +d=interactiveCorba parameter.

The test code we downloaded last time came with rather a lot of files, most of which we don't need when creating a new script, so let's set up a minimal example. First, create a directory for this new script.

cd $PREFIX/omniORB
mkdir -p apps/idl_example
cd apps/idl_example


Next, copy omc_communication.py and omc_communication.idl into this directory.

cp $PREFIX/omniORB/src/python_2.5-omniORBPy_3.3-win32/omc_communication.{py,idl} .

These are the only two files we need to get started. The next step is to compile the IDL file to provide your script with the appropriate interfaces. The default method of doing this creates two folders and an additional file in your directory, which is a little too messy for me. To prevent this, you can tell omniidl (the omniORB IDL compiler) to place all the files into a package. The documentation has a warning about this.

Note that if you use these options to change the module package, the interface to the generated code is not strictly-speaking CORBA compliant. You may have to change your code if you ever use a Python ORB other than omniORBpy.

However, we're only using omniORBpy here, and the files are easy enough to regenerate if you ever reconsider. With that in mind, let's compile the IDL.

omniidl -bpython -Wbpackage=idl omc_communication.idl

You should now see an idl directory containing:
  • _GlobalIDL
    • __init__.py
  • _GlobalIDL__POA
    • __init__.py
  • __init__.py
  • omc_communication_idl.py

Because we compiled the IDL a little differently to the default, we need to edit the test script. Simply change line 6 to read:

import idl._GlobalIDL as _GlobalIDL

Now run the script:

python omc_communication.py

If all is well, you should see the same graph as in part 1.

Wednesday, October 30, 2013

OpenModelica and Python via CORBA (Part 1: getting started)

Part 1 (getting started) | Part 2 (IDL compilation) | Part 3 (Python wrapper)


Note: It is very likely that the process described in this series of posts has already been performed by someone else. For example, a Python wrapper for OMC's CORBA interface has been implemented by a team in Sweden whose code is already included with OpenModelica. However, this series should hopefully provide sufficient information to allow you to implement such a wrapper in a different language that doesn't already have one. From a brief search, C++ seems to be one such that also happens to be supported by OmniORB. Java provides its own CORBA libraries but does not appear to have a convenient OMC CORBA wrapper yet.


In this post I'm going to be walking through the process of setting up omniORB with Python for the purpose of interacting with the CORBA interface provided by the OpenModelica compiler (OMC). This allows OMC to be controlled from a Python script (loading simulations, changing parameters, etc). While the intention of this post is specifically that of interacting with OMC, the installation of omniORB will be the same in other situations. Note though that you may require additional setup in these other situations, for omniNames for example. I will be showing the setup process for Linux (and specifically, Mint/Ubuntu). I will assume that OpenModelica is already installed.

The first step is to download the latest omniORB and omniORBpy releases from http://sourceforge.net/projects/omniorb/files/. At the time of writing these were omniORB 4.1.7 and omniORBpy 3.7. I'm going to assume you want to install to your home directory rather than to a system-wide location.

Open up a terminal and setup the root directory omniORB (replace user with your own username).

mkdir -p omniORB/src
cd omniORB/src

export PREFIX=/home/user/omniORB


Extract the previously downloaded files to the src directory, making sure to preserve directory structure. After that, build omniORB and omniORBpy.

cd omniORB-4.1.7
mkdir build
cd build
../configure --prefix=$PREFIX
make
make install


cd ../../omniORBpy-3.7
mkdir build
cd build
../configure --prefix=$PREFIX
make
make install


Your $PYTHONPATH environment variable needs to be extended to include the Python module install location and your $PATH variable needs to include the omniORB bin directory. I haven't made these edits permanent, so they must be done each time the Python modules or the utilities are required in a new shell.

export PYTHONPATH=$PYTHONPATH:$PREFIX/lib/python2.7/site-packages
export PATH=$PATH:$PREFIX/bin


At this point, you have a functional install that will allow you to interface to the OpenModelica compiler from a Python script. The remainder of this post uses some existing example code to test that everything is working correctly.

First, download the code from http://www.ida.liu.se/~adrpo/omc/corba/_windows/python_2.5-omniORBPy_3.3-win32.zip. This was written for Windows, so there are a few changes to make.
  • Add to the list of imports:
    • import getpass
  • Edit line 15:
    • omc_objid_file = tempdir + "/" + "openmodelica.%s.objid" % getpass.getuser()
  • Edit line 45:
    • "loadFileInteractive(\"/usr/share/doc/omc/testmodels/BouncingBall.mo\")",

In a different terminal run omc with the CORBA interface active. You could run this in the background in the current terminal, but I prefer to be able to distinguish between output from different programs.

omc +d=interactiveCorba

Back in the original terminal:

export TEMP=$PREFIX/tmp
python omc_communication.py


If the test is successfull you should see a graph as below.

Wednesday, October 23, 2013

Python: generator-based zip

Update:
It has come to my attention that this already exists: izip in the itertools module provides the same functionality. I will leave this post up to help in pointing people to a solution.

One thing that has bitten me occasionally in the past is that the zip function in Python assembles the entire output list when the call is made. This means that when dealing with very large lists the call can take a significant amount of time.

My solution to this was to implement a generator-based zip function that only deals with one element from each source at a time. The implementation is simply:

def zipgen(*args):
    """
    A generator-based implementation of the zip function. Unlike zip,
    this does not pre-generate the complete output lists.
    """
    iters = [x.__iter__() for x in args]
    while 1:
        yield [x.next() for x in iters]

Hopefully this will be helpful if you're ever in a similar situation.