geoptima

== Welcome to geoptima.rb

Geoptima is a suite of applications for measuring and locating mobile/cellular subscriber experience on GPS enabled smartphones. It is produced by AmanziTel AB in Helsingborg, Sweden, and supports many phone manufacturers, with free downloads from the various app stores, markets or marketplaces. This Ruby library is capable of reading the JSON format files produced by these phones and reformating them as CSV for further analysis in Excel. This is a simple and independent way of analysing the data, when compared to the full-featured analysis applications and servers available from AmanziTel. If you want to analyse a limited amount of data in excel, or with Ruby, then this GEM might be for you. If you want to analyse large amounts of data, from many subscribers, or over long periods of time then rather consider the NetView and Customer IQ applications from AmanziTel at www.amanzitel.com.

More recent versions of this script, since about 0.1.0, have been able to perform statistics and generate charts. See below for the csv_chart script usage.

For more information on Geoptima refer to:

• {AmanziTel}[http://www.amanzitel.com] - for commercial applications and platforms supporting Geoptima
• {Geoptima on Android}[https://market.android.com/details?id=com.AmanziTel.Geoptima] - on the android market
• For Geoptima on other plaforms, use the phone to search the relevant app-store or market

=== Documentation

The current version includes the core geoptima.rb library as well as a few command-line apps for using the library. We have not yet documented the library, but some of the command-line scripts are described below.

==== show_geoptima

This script imports the JSON files on the command-line and then:

• Prints out basic information about each file (subscriber information, start date and number of events)
• Optionally prints out all events on the console for visual inspection (using the -p option)
• Optionally prints out all events to files in CSV format with merging of event types and fields into a single header for further processing with Excel or OpenOffice (using the -x option)
• The CSV export mode has many further options for exporting to multiple files (-s), merging IMEI's into single dataset (-a), long headers with repeated information (-l), etc.
• Configuration of the export format (static columns and header names) is possible with the -l and -M options

Which event types to include and various other options are available using the command-line. Run it with the -h option to get a full list of options. The current version should support:

Usage: show_geoptima <-dwvpxomlsafegh> <-P export_prefix> <-L limit> <-E types> <-T min,max> <-M mapfile> file -d debug mode (output more context during processing) (false) -w verbose mode (output extra information to console) -v print geoptima library version 0.1.19 -p print mode (print out final results to console) -x export IMEI specific CSV files for further processing -o export field statistis -m map headers to classic NetView compatible version -l longer header list (phone and operator fields) -s seperate the export files by event type -a combine all IMEI's into a single dataset -f flush stdout -e show error statistics -g export GPX traces -t split time colum to multiple columns -h show this help -P prefix for exported files (default: ''; current: ) -E comma-seperated list of event types to show and export (default: all; current: ) -A application category map file (default: app names) -T time range to limit results to (default: all; current: ) -B location limited to specified bounds in one of these formats: minlat,minlon,maxlat,maxlon minlat..maxlat,minlon..maxlon DIST(distance_in_km,lat,lon) RANGE[minlat,minlon,maxlat,maxlon] RANGE[minlat..maxlat,minlon..maxlon] (default: all; current: ) -L limit verbose output to specific number of lines (10000) -M mapfile of normal->altered header names: -G GPX export options as ';' separated list of key:value pairs Current GPX options: {"scale"=>190, "padding"=>5, "limit"=>2, "png_limit"=>10, "points"=>true, "point_size"=>2, "point_color"=>"auto"} -X Geolocation options as ';' separated list of key:value pairs Current geolocation options: {"algorithm"=>"window", "window"=>60}

The GPX and Geolocation options require futher explanation:

Known supported GPX options (might be more, see data.rb code): limit:2 Limit GPX output to traces with at least this number of events png_limit:10 Limit PNG output to traces with at least this number of events merge: Merge all traces into a single trace only_merge: Do not export unmerged traces scale:190 Size of print area in PNG output padding:5 Space around print area points:true Turn on/off points point_size:2 Set point size point_color:auto Set point color: RRGGBBAA in hex (else 'auto') format: Export format: 'gpx', 'csv', 'png', default 'all' waypoints: Export waypoints for events: <event_type>, default 'all'

PNG images will be 'scale + 2 * padding' big (200 for current settings). The scale will be used for the widest dimension, and the other will be reduced to fit the actual size of the trace. No projection is used, with the points simply mapped to their GPS locations. This will cause visual distortions far from the equator where dlat!=dlon.

Known supported geolocation options (might be more, see data.rb code): algorithm:window Which geolocation algorithm to use window:60 Time window in seconds, has slightly different meanings for different algorithms

Currently supported algorithms: window: select GPS point within window seconds of event, GPS points after the event take priority. (this is the default for geoptima GEM version >= 0.1.19) +win: select only GPS points after event (within time window) -win: select only GPS points before event (within time window) (this is the default for geoptima GEM version < 0.1.19) closest: select closest GPS point within window seconds of event (similar to window option, but chooses closest) interpolate: Linear interpolation between two closest points (experimental, do not use yet) (read redmine wiki page for explanation of algorithm)

Currently the script also locates events that are close enough in time to GPS events. We hope to improve this with interpolation in the near future to be more compatible with the results from the commercial solutions. This time-window is also used for some of the extended header information, like LAC and CI, and effectively duplicates those fields from their own events to others. Take this into account when doing statistics on the results. It is better to use the original values, not the duplicates, if you want reliable statistics.

==== csv_stats

This script reads any CSV file and outputs histograms of the values for each column. These can be manually imported into a spreadsheet or charting program for further analysis or charting. This script is to a large extent superceded by the csv_chart script which both calculates stats as well as generates charts of the results.

==== csv_charts

This script also performs statistics on CSV files but is much more powerful, including automatic output of charts based on columns of data, as well as a configurable system for adding new columns by modifying or merging other columns, as designing specific charts for output.

Usage: csv_chart <-dhamt> <-S specfile> <-N name> <-D dir> <-T range> <-P diversity> files... -d debug mode -h print this help (true) -a automatically create charts for all properties (true) -m merge all files into single stats -t merge and split by time (days) -N use specified name for merged dataset: -D export charts to specified directory: . -S use chart specification in specified file: -P diversity threshold in percentage for automatic reports: 40.0 -T set time-range filter:

The -S option is especially interesting, since you can control both custom columns and chart output. Consider the following examples:

Signal strength in 10dBm category ranges bar chart

stats 'Signal Strength', 'RSSI', :div => 10 histogram_chart 'Signal.Strength'

Multi-column chart for different event types over days of the week

stats 'Call Status', 'call.status', :group => :days stats 'Data Status', 'data.status', :group => :days stats 'Events', 'Event', :group => :days category_chart 'Data Status' category_chart 'Events' category_chart 'Call Status'

The same stats as before but in line charts

line_chart 'Data Status' line_chart 'Events' line_chart 'Call Status'

Top-10 charts for very diverse values like LAC-CI

stats('URL', 'browser.URL') {|v| v && v.gsub(/%(\w{2})/){|m| $1.to_i(16).chr}[0..50]} stats('Apps', 'runningApps.appName', 'runningApps.state') {|name,state| (state=='STARTED') ? name : nil} histogram_chart 'LAC-CI', :top => 10, :side => true histogram_chart 'Event', :top => 10, :side => true histogram_chart 'URL', :top => 10, :side => true histogram_chart 'Apps', :top => 10, :side => true

One thing to be careful above with the stats command is that if you do not provide a block, the part of the line in {}, then you also do not need brackets. However, if you provide the block, then you must also use brackets. See the examples above for both cases.

The contents of the brackets are the options to the command, and the block in {} is a piece of code to be run when converting the input columns to the output columns. If you pass three arguments to the stats command, the first is the new column name, the next two are the columns to use, and these two will be passed as two parameters to the block. Let's look at three examples:

stats 'Data Status', 'data.status'

This simply creates a copy of the column 'data.status' with the new name 'Data Status', so that resulting charts have a nicer title. No block is given, and the identity operator is assumed. This is the same as passing the block {|v| v}.

stats 'Data Status', 'data.status', :group => :days

The extra option to group results in several columns being made, one for each day for which data was found. This allows for the data to be charted as multiple dataset on the same chart. The default grouping behaviour here is to have each day on the x-axis, and each value as a new dataset.

stats 'Signal Strength', 'signal.strength', :div => 10

The div option will cause values within a range of 10 to be merged into a new value with the name of the range. In this case a value like -53 will become -60..-50. This is useful for making bar charts with fewer bars.

stats('URL', 'browser.URL') {|v| v && v.gsub(/%(\w{2})/){|m| $1.to_i(16).chr}[0..50]}

This example uses some fancy Ruby code to replace cryptic HTTP URL codes with normal ASCII, so the URL's look better.

stats('Apps', 'runningApps.appName', 'runningApps.state') {|name,state| (state=='STARTED') ? name : nil}

This one copies the 'runningApp.appName' column to the 'Apps' column, but only if the 'runningApps.state' column is set to 'STARTED'.

=== Charts

The charts described above can also be seen as examples in the Gallery wiki page at https://github.com/craigtaverner/geoptima.rb/wiki/Gallery.

One example included here is a distribution of call status over time.

https://github.com/craigtaverner/geoptima.rb/blob/master/images/Chart_All_Call%20Status_category_distribution.png?raw=true

=== Installation

Two options:

• As a RubyGem: gem install geoptima json
• From source: git clone [e-mail]:craigtaverner/geoptima.rb.git

If you install without the json gem you will get a working library but not well optimized for JSON parsing. To install the json gem you need the minimal build environment for installing C code gems. On Linux and Mac this is usually already the case (or fix with 'apt-get install build-essential'). On windows you need to install the DevKit. See below for windows specific installation instructions.

When installing the ruby gem you get the commands like 'show_geoptima' and 'csv_charts' in the path. However, if you install from source, you instead get them as examples/show_geoptima.rb.

git clone [e-mail]:craigtaverner/geoptima.rb.git cd geoptima.rb/examples ./show_geoptima.rb -x sample_geoptima.json

This should produce a file called 357841036600753.csv containing a CSV version of the Geoptima events for importing into Excel or OpenOffice.Calc for further processing. The number is the IMEI of the phone used, and if you process many JSON files in one go, you will get as many output files as there are phones represented by the data.

The examples directory also contains a number of sample *.spec files for the csv_chart command. In order for csv_chart to be able to generate charts, you need to also install a supported chart library. We have planned support for various charting libraries, but at the moment only support GRUFF. For installation of GRUFF, see below.

==== Installing on Windows

For windows use the RubyInstaller at http://rubyinstaller.org/

To get the optimized json gem installed, or to get chart support working you will need to install the DevKit as described on https://github.com/oneclick/rubyinstaller/wiki/Development-Kit. Please follow those instructions carefully as they are not entirely trivial.

==== Running on Windows

The best way to run on windows is to edit your system path (usually go to 'my computer->properties->advanced->environment), and add the path to your ruby bin directory (usually C:\Ruby\bin) to the end of the system PATH. After that, open a command-prompt (start->cmd.exe) and type 'ruby -v' to see that it worked. Then you can execute the 'gem install' commands and the 'show_geoptima' commands from this console.

==== Installing Gruff for Charting

In order to support charting, you need to install Gruff (see http://nubyonrails.com/pages/gruff). This depends on image-magick, so you need to install that first. Please refer to the installation instructions specific to your OS. On Ubuntu the following should work: apt-get install imagemagick libmagickcore-dev gem install rmagick

=== Contributing

Have you found a bug, need help or have a patch ? Just clone geoptima.rb and send me a pull request or email me.

=== License

MIT, see the LICENSE file http://github.com/craigtaverner/geoptima.rb/tree/master/LICENSE.