Category Archives: Research

Install GDAL on Mac OS X (Mavericks)

I have Quantum GIS on my Mac.

I also used to have GDAL – or rather, I do have GDAL, because QGIS does not work without GDAL?

However I’ve been unable to use GDAL recently.  So, I went to install it again. 

The first time, I think I went through Kyngchaos. Which was not easy. 

The second time, now, I’m going to use Homebrew. Because it is easier.  

Actually it’s outlined here: https://trac.osgeo.org/gdal/wiki/BuildingOnMac

Here’s what I did:

tjessel:~ apple$ brew install gdal

Then it does all this stuff:


 

==> Installing dependencies for gdal: giflib, lzlib, proj, libgeotiff, ge

==> Installing gdal dependency: giflib

==> Downloading https://downloads.sourceforge.net/project/giflib/giflib-4.x/gifl

######################################################################## 100.0%

==> ./configure –prefix=/usr/local/Cellar/giflib/4.1.6

==> make install

🍺  /usr/local/Cellar/giflib/4.1.6: 39 files, 720K, built in 69 seconds

==> Installing gdal dependency: lzlib

==> Downloading http://download.savannah.gnu.org/releases/lzip/lzlib-1.5.tar.gz

######################################################################## 100.0%

==> ./configure –prefix=/usr/local/Cellar/lzlib/1.5 CC=clang CFLAGS=

==> make

==> make check

==> make install

🍺  /usr/local/Cellar/lzlib/1.5: 9 files, 152K, built in 8 seconds

==> Installing gdal dependency: proj

==> Downloading https://downloads.sf.net/project/machomebrew/Bottles/proj-4.8.0.

######################################################################## 100.0%

==> Pouring proj-4.8.0.mavericks.bottle.tar.gz

🍺  /usr/local/Cellar/proj/4.8.0: 47 files, 5.7M

==> Installing gdal dependency: libgeotiff

==> Downloading http://download.osgeo.org/geotiff/libgeotiff/libgeotiff-1.4.0.ta

######################################################################## 100.0%

==> ./configure –prefix=/usr/local/Cellar/libgeotiff/1.4.0 –with-libtiff=/usr/

==> make

==> make install

🍺  /usr/local/Cellar/libgeotiff/1.4.0: 70 files, 5.8M, built in 37 seconds

==> Installing gdal dependency: geos

==> Downloading http://download.osgeo.org/geos/geos-3.4.2.tar.bz2

######################################################################## 100.0%

==> ./configure –prefix=/usr/local/Cellar/geos/3.4.2

==> make install


 

The installation stopped here for a while. 

 3:32 PM. Seems to be hung up.  Maybe I should give it some time? Seems to have finished up at 3:42 or so.

I’m pasting the rest here: 


 

🍺  /usr/local/Cellar/geos/3.4.2: 385 files, 8.4M, built in 10.4 minutes

==> Installing gdal dependency: freexl

==> Downloading http://www.gaia-gis.it/gaia-sins/freexl-1.0.0g.tar.gz

######################################################################## 100.0%

==> ./configure –prefix=/usr/local/Cellar/freexl/1.0.0g

==> make install

🍺  /usr/local/Cellar/freexl/1.0.0g: 10 files, 144K, built in 25 seconds

==> Installing gdal dependency: libxml2

==> Downloading ftp://xmlsoft.org/libxml2/libxml2-2.9.1.tar.gz

######################################################################## 100.0%

==> ./configure –prefix=/usr/local/Cellar/libxml2/2.9.1 –without-python

==> make

==> make install

==> python setup.py install –prefix=/usr/local/Cellar/libxml2/2.9.1

creating build/temp.macosx-10.9-intel-2.7

clang -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -mno-fused-madd -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe -D_REENTRANT=1 -I/usr/local/Cellar/libxml2/2.9.1/include/libxml2 -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/include -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c libxml2-py.c -o build/temp.macosx-10.9-intel-2.7/libxml2-py.o

clang: error: unknown argument: ‘-mno-fused-madd’ [-Wunused-command-line-argument-hard-error-in-future]

clang: note: this will be a hard error (cannot be downgraded to a warning) in the future

error: command ‘clang’ failed with exit status 1

Error: Homebrew doesn’t know what compiler versions ship with your version

of Xcode (5.1.1). Please `brew update` and if that doesn’t help, file

an issue with the output of `brew –config`:

  https://github.com/Homebrew/homebrew/issues

Note that we only track stable, released versions of Xcode.

Thanks!

READ THIS: https://github.com/Homebrew/homebrew/wiki/troubleshooting


 

So that was a lot easier than I remember the Kyngchaos method… I hope that this works. I think I need to check the version. 

I think this is how you do it. 

http://www.gdal.org/gdal-config.html

It’s possible I missed a key step, because I typed “gdal-config –version” and did not get anything back… I dug through some old notes and found a link to this site:

https://sites.google.com/site/eospansite/installinggdalonmacosx

Which said a final step was to open the Terminal and type in:

export PATH=/Library/Frameworks/GDAL.framework/Programs:$PATH

Seems to have worked; I typed in gdal-config –version and got a result:

tjessel:~ apple$ gdal-config –version

1.10.1

Hopefully this means my GDAL installation is working again!

Advertisements

U.S. Virgin Islands Benthic Habitat Spatial Data

Web Site Owner: National Ocean Service

I am enamored of this dataset; it is essentially “marine ecological regions” for U.S. territorial holdings in the Carribean (Puerto Rico, U.S. Virgin Islands):

http://ccma.nos.noaa.gov/products/biogeography/benthic/data

Along with simply satisfying curiosity, I think it might be useful for understanding the spatial distribution and preferred habitat of invasive lionfish in the Virgin Islands, in a broad sense.

Pairing occurrence records with habitat classifications may provide useful insights.  There is a Java application called Weka that can reveal clustering in your data.

By pairing this data with the names of the delineated habitat, I believe the tabular data within the spatial dataset can reveal any natural clustering (Although the clustering may be biased towards popular dive / snorkeling sites).

Another interesting data point which clustering can reveal is the depth at which the animals are most often found. However, this again may simply be a reflection of the “preferred depth” of divers (e.g., not greater than 30 m).

Aerial frame for Sandy Point NWR: http://www8.nos.noaa.gov/biogeo_public/aerial/viewer.aspx?source=usvi&id=244

Blacklight UI for PostgreSQL and Solr – Google Group Discussion

A discussion thread I started in January, 2014: https://groups.google.com/forum/#!topic/blacklight-development/Ax0840hBeUA

Hello,

I’ve poked around a bit and did not see anything about support for geospatial search.
I became interested in Blacklight after visiting the Environment Australia Web site.
Their site is built on PostgreSQL relational database, and since it pertains to the environment, I thought they might have spatially explicit data.
They do have “Search by Region,” but it is just a linked list of regions.
Thanks for any tips,
Tanner

I got a few responses asking for more information and potential use cases so I added (in July 2014):


My opinion is any geographic representation of a collection constitutes value-added.

I am interested building a functional SKOS ontology based on the relationships between EPA ecological regions and protected areas of the United States. There are a few hierarchical levels which the ontology could describe. There are also relationships between protected areas and each ecological region.

For example, Great Smoky Mountains National Park on the North Carolina and Tennessee belongs to two main ecological regions: Ridge and Valley (67) ,and Blue Ridge (66). A full accounting of Level III ecological regions is available at <ftp://ftp.epa.gov/wed/ecoregions/cec_na/NA_LEVEL_III.pdf>.

The Level III ecological regions can be further divided into Level IV ecoregions, a higher level of granularity, available at <ftp://ftp.epa.gov/wed/ecoregions/us/Eco_Level_IV_US.pdf>.

Using the Smokies again as the exemplar, new ecological sub-regions at the Level IV resolution emerge: High mountains (66i), Southern sedimentary ridges (66e), Limestome Valleys and Coves (66f), Southern Metasedimentary mountains (66g), Broad basins (66j), and Southern Dissected Ridges and Knobs (67i), Southern sandstone ridges (67h), Southern Shale Valley (67g).

These regions can be linked via an ontology to other protected areas, allowing environmental information resources to be grouped based on meaningful ecological relationships. For example, a search in an information retrieval system for “High Mountains (66i)” would retrieve results from relevant protected areas: Roane Mountain State Park, Pisgah National Forest, Cherokee National Forest, AND Great Smoky Mountains National Park, and perhaps any location built in to the ontology as a member of the given ecological region’s footprint. This represents a sophisticated query with minimal effort on behalf of the user.

Along with providing a framework for information retrieval via “regions” as the Australia site does, the ontology would have useful text mining and automated spatial metadata creation applications.

Potential use cases include:

I wanted to do a Masters thesis on the impact of an ontology on search and retrieval, but after discussing with my thesis coordinator who indicated this was more of a PhD level undertaking, I opted to instead pursue comprehensive exams as my exit strategy.

I still think this is a worthwhile area of research and I am happy to see the example you have shared, and that there are others interested in the topic.

Bookmarklets

I use a variety of bookmarklets that make life easier.

Bookmarklets are little pieces of javascript that you can apparently drag to your browser’s toolbar.

These days, I’m using Chrome because I heard that Safari has some security vulnerabilities.

So, I’ve re-installed all my bookmarklets and realized they were hard to find.

To keep from losing them again, I’m listing them here:

WordPress “Press This” Bookmarklet – available from your “Tools” page (e.g.,

https://mountainsol.wordpress.com/wp-admin/tools.php)

Press This is a bookmarklet: a little app that runs in your browser and lets you grab bits of the web.

Use Press This to clip text, images and videos from any web page. Then edit and add more straight from Press This before you save or publish it in a post on your site.

Twitter Share Bookmarklet:

I don’t use this a lot because I tend to post things in Google + if I want them to go to twitter. I have a IFTT setup for syncrhonizing G+, Twitter, and Facebook to publish in that direction.

https://dev.twitter.com/docs/share-bookmarklet

Google Blogspot/Blogger Bookmarklet

I use this one for a few of my Google Blogger blogs.

https://support.google.com/blogger/answer/41469?hl=en

Tumblr Bookmarklet

This is for my “Data Pro” tumble blog. It’s now available at <https://www.tumblr.com/apps&gt;.

UT Libraries Off-campus Proxy Bookmarklet

http://www.lib.utk.edu/databases/remoteaccess/proxybook.php

Save to Mendeley Bookmarklet:

http://www.mendeley.com/import/

Others:

I also use the Pinterest extension on Chrome for saving interesting images.

I use LastPass extension.

Embedding Instructional Material Hosted Elsewhere

I’m interested in the instructional material in data curation hosted by ESIP Federation online at <http://wiki.esipfed.org/index.php/Data_Management_Course_Outline&gt;.

I searched Google for “Online Powerpoint Viewer” and located this: <https://docs.google.com/viewer&gt;.

I think it could be useful for enabling embedded instructional material without using a service like SlideShare, a service that requires one own the material uploaded.

Take the first URL from the ESIP Federation: http://wiki.esipfed.org/images/6/6d/AgencyRequirementsV2.ppt

Add the URL to Google’s little viewer and you get two pieces of code to choose from:
Link to viewer:
[<a href="http://docs.google.com/viewer?url=http%3A%2F%2Fwiki.esipfed.org%2Fimages%2F6%2F6d%2FAgencyRequirementsV2.ppt">Agency requirements </a>]
Embedded on Web site:
googleapps domain="docs" dir="viewer" query="url=http%3A%2F%2Fwiki.esipfed.org%2Fimages%2F6%2F6d%2FAgencyRequirementsV2.ppt&embedded=true" width="600" height="780" /
Now I will try the “Embedded on a Web site” code above in the space immediately below the horizontal rule:


This Drupal module <https://www.drupal.org/project/groupdocs_viewer> is something I’d like to try for a Drupal Website; I have a “sandbox” drupal site to play with at datapro.drupalgardens.com.  At some point I will explore how to accomplish embedding powerpoint documents in a Drupal Website or perhaps a wiki, such as my OpenWetWare research notebook for technical documentation of Maxent at <http://openwetware.org/wiki/GSMNP:Notebook/MaxEnt>.  If it’s possible to embed a powerpoint presentation on a wiki that is hosted by the same wiki, that could be useful for my IS 599 Practicum Experience work.

Eastern Hemlock (Tsuga canadensis) model on PC versus Supercomputer

I recently ran a model of Hemlock on my personal computer.

I modeled ~489 records; the Nautilus model used over 2,000.

From the model output:

This is a representation of the Maxent model for Tsuga_canadensis. Warmer colors show areas with better predicted conditions. White dots show the presence locations used for training, while violet dots show test locations.

 

This slideshow requires JavaScript.

This is not really a fair comparison, but the difference between the models with 489 records and 2000+ is interesting for comparing the predictions.

EECS model for Eastern hemlock is different.  It uses more data, and it was run 20 times with 10% of the records reserved before being synthesized into one image.

I will post some additional comparisons for other trees in some later posts.

Possible Spatial Data Inputs for MaxEnt Species Distribution Models in Great Smoky Mountains N.P.

Environmental layers are available to the public via IRMA.

Source metadata are not available at http://tiny.utk.edu/atbi.

I have attempted to map or cross-walk the layers listed by the Simmerman et. al paper to the names of datasets available for download from IRMA.

Table. Mapping from UTK names to IRMA names.

1 Soil Organic Type Soil Classification https://irma.nps.gov/App/Reference/Profile/2198007
2 Topographic Convergence Index Topographic Wetness Index https://irma.nps.gov/App/Reference/Profile/2208650
3 Solar Radiation Data 30 -m Potential Solar Radiation https://irma.nps.gov/App/Reference/Profile/2208716
4 Terrain Shape Index 30-m Topographic Shape Index https://irma.nps.gov/App/Reference/Profile/2208684
5 Terrain Shape Index 30-m Topographic Ruggedness Index Model https://irma.nps.gov/App/Reference/Profile/2182017
6 Digital Elevation Model 30-m Lidar Digital Elevation Model https://irma.nps.gov/App/Reference/Profile/2180606
7 Slope in Degrees 30-m Lidar Slope Model https://irma.nps.gov/App/Reference/Profile/2180632
8 Understory Density Classes Understory Vegetation at GRSM https://irma.nps.gov/App/Reference/Profile/1047499
9 Leaf On Canopy Cover Overstory Vegetation at GRSM https://irma.nps.gov/App/Reference/Profile/1047498
10 Vegetation Classes Vegetation Classification Great Smoky Mountains NP Vegetation Classification https://irma.nps.gov/App/Reference/Profile/1021458

Note: I am grateful to http://www.textfixer.com/html/csv-convert-table.php which made it possible to easily create this table from plain text.  I expect to add this to my “toolkit” of useful items and it saved me a lot of time.

Spatial Data Diversity Supporting Herpetofaunal Research in Great Smoky Mountains National Park

2014 North Carolina PARC Poster.

Reduced file size image of poster presented at North Carolina Partners in Amphibian and Reptile Conservation (NCPARC) meeting, March 2014.

Poster presented at 2014 North Carolina Partners in Amphibian and Reptile Conservation Meeting.

              Jessel, Tanner; Super, Paul E.; Colson, Thomas (2014): Spatial Data Diversity Supporting Herpetological Research in Great Smoky Mountains National Park. figshare.

http://dx.doi.org/10.6084/m9.figshare.978500

SDM re-projected for Google Earth, OSM with gdal2tile

I think I have stumbled upon the solution for tiling the png image.
To re-project the PNG image, we can geo-reference the images with GDAL, then warp the geo-referenced image to the correct projection, also with GDAL.
The process is described here:
First, enter “gdalinfo Abies_fraseri.png” into terminal to get the bounds of the PNG image.
This yields the following output:
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 1302.0)
Upper Right ( 2899.0,    0.0)
Lower Right ( 2899.0, 1302.0)
Center      ( 1449.5,  651.0)
A template and implementation for our PNG files is demonstrated here:
Template:
gdal_translate -of VRT -a_srs EPSG:4326 -gcp 0 0 ULlong ULlat -gcp UPPERRIGHTPx 0 URlong URlat -gcp LOWERRIGHTPx LOWERRIGHTPy LRlong LRlat Abies_fraseri.png Abies_fraseri.vrt
Implementation: 
gdal_translate -of VRT -a_srs EPSG:4326 -gcp 0 0 -84.000683874 35.7889383688 -gcp 2899.0 0 -83.0424855 35.7889383688 -gcp 2899.0 1302.0 -83.0424855 35.426963641 Abies_fraseri.png Abies_fraseri.vrt

For “ULlong” (Upper Left Long) and so forth I used a bounding box tool (http://boundingbox.klokantech.com) to determine the following bounds of the PNG – however, if there is a more “official” known boundary, it would be wise to use that instead.
                                35.426963641,-84.000683874 – bottom left / lower left
                                35.426963641,-83.0424855 bottom right / lower right
                                35.7889383688,-83.0424855 – top right / upper right
                                35.7889383688,-84.000683874 – top left / upper left
Next, take the .vrt file and warp it:
    gdalwarp -of VRT -t_srs EPSG:4326 Abies_fraseri.vrt Abies_fraseri_2.vrt
This creates a folder with tiles for a KML network link.
see “doc.kml” in the attached zip folder which you may open in Google Earth.
There is also an Open Map Layers HTML page in the attached zip folder.
To add trails in Google Earth
My opinion is it would be more interesting and informative to have the species occurrences added as a separate layer.  Therefore, I would prefer to create placemarkers / waypoints for species locations rather than black squares on the images.  Is it possible to generate PNG images with the likely distribution but not the black squares?  If so, would that take a long time to re-do to make these tiled overlays?
Also, although I “hacked” the PNG bounds, I’m still worried it the overlay’s georeferencing be “off” a bit and would love to use a more trustworthy set of coordinates than those derived from my best judgement and http://boundingbox.klokantech.com.
From here, I think it would be smart to document a workflow for doing a lot of PNG to VRT to KML at a time.  I haven’t tried processing a whole directory of images at once yet – just playing with the Abies_fraseri.png for now.
The doc.kml and virtual raster dataset work with Google Map API and Open Layers map.  I think they both run on Javascript.

Environmental Layers for HPC Maximum Entropy Species Distribution Models in Great Smoky Mountains National Park

Currently there are 10 environmental layers that were used by Simmerman et. al in the demonstration project, “Exploring similarities among many species distributions.”

  1. Bedrock geology
  2. Digital elevation model
  3. Slope measured in degrees
  4. Solar radiation data
  5. Soil organic type
  6. Terrain shape index
  7. Topographic convergence index
  8. Leaf on canopy cover
  9. Understory density classes
  10. Vegetation classes

This slideshow requires JavaScript.

The contribution of environmental variables to each “Maximum Entropy” Species Distribution Model (MaxEnt SDM) environmental variables are accessible from each model under the “Environmental Layers” tab.

The help icon accompanying each model provides this text:

This is a species distribution model (SDM) produced by MaxEnt. This SDM is actually a composite from ten cross-validation runs for each species (see cross-validation results tab for more information). Original ATBI record locations are shown with black dots. The color scale goes from 0 probability of presence (dark brown) to 100% probability (dark green).

Color distribution depicting a probability between 0 (no chance of finding a species) and 1 (100% chance of finding a species)

Color distribution depicting a probability between 0 (no chance of finding a species) and 1 (100% chance of finding a species)

Below is a screen capture of the model generated for the Fraser fir (Abies fraseri). This model is based on 474 occurrence records.  The full size image is available online at <http://seelab.eecs.utk.edu/alltaxa/maps/Abies_fraseri.png>

Abies_fraseri

SDM model generated by the ATBI mapping project / University of Tennessee SEElab.

Interestingly, the MaxEnt model output suggests that the “Digital Elevation Model” contributed 88.8% to the Species Distribution Model for Abies frasieri.  This makes sense, since the Fraser fir is a species of conifer favoring cold environments that inhabits only the highest elevations of the Park.  The remaining layers contribute less than 5% to the model.

I’m copying out the taxonomic classification from Wikipedia:

Scientific classification
Kingdom: Plantae
Division: Pinophyta
Class: Pinopsida
Order: Pinales
Family: Pinaceae
Genus: Abies
Species: A. fraseri

The purpose is to access the records from the ATBI database, where I don’t see the Fraser fir listed in the ATBI “plants” kingdom. From <http://tremont22.campus.utk.edu/ATBI_Query.cfm> I searched by “order’ for “pinales.”

Fraser fir is accessible: <http://tremont22.campus.utk.edu/ATBI_Species.cfm?genus=Abies&epithet=fraseri&subspecies=%7E>.  Interestingly, the number of specimens in the database is 866.  Contrast this with the 474 records that were used to generate the model.  The model may simply be older (there does not appear to be a timestamp for the model; either from “Get Info” or opening up “Properties” in GIMP for both the large 299×1302 pixel PNG file and the small 600×269 pixel png) or, not all 866 occurrence records have spatial coordinates associated (e.g., they are “references from the literature.”)