Geoprocessing In ArcGIS

Geoprocessing is a cornerstone of ArcGIS, and is a general term that describes most of the spatial analytic functions of the program.  In my previous jobs working with GIS I didn't really use these tools, as I was primarily making maps for visual display. The use of GIS in any capacity beyond this typically requires use of geoprocessing tools.

The graphic above is a polygon representation of a basin area, with areas containing soil types designated as "not suitable for farming" removed. This was derived from polygon vector graphics of the basin and the various soil types that this area consists of, but the notable aspect of it is that the operation to remove the specific areas with the farming-unsuitable soil was performed with a Python script/model within ArcGIS.  Normally, one might use each tool within the program- clip, select and erase- one at a time, with the end result being the polygons above.  To streamline the workflow, which would be necessary for dealing with data in greater volumes, we can create a model within ArcGIS which will perform each operation in sequence when it is run, and can also be exported and modified as a Python script.  Models exported and modified as scripts have the advantage of potentially performing more complicated analyses, can work without the necessity of having ArcMap open, and can be scheduled to run at a designated time, automatically.  In my previous GIS career at a city board of public works, which administered water, sewer and electricity services, I worked with someone who created these kinds of scripts for our daily use, and they were essential to our operation.  I feel fortunate to now have the opportunity to learn more about how that might be done, and hope to possibly provide something as useful to someone's workflow someday myself.  

Visibility Analysis and Camera Surveillance

Modern society, in public places, is under nearly constant video surveillance.  (http://www.cnn.com/2013/04/26/tech/innovation/security-cameras-boston-bombings/) The merits and ethics involved in public video surveillance are endlessly debatable, but the logistic issues inherent in constant camera visibility do present some uniquely challenging problems involving spatial analysis.  Finding the best locations and viewing angles for cameras capturing the finish line of the Boston Marathon is the task du jour, aided by use of the 3-dimensional spatial analytic tools available in ArcGIS.

Above is the location in Boston where the finish line of the marathon is placed- symbolized with a pink star.  The purple circles are the camera locations, and the shades of blue and green are locations visible from one, two and all three of the cameras.  The cameras are placed to maximize the viewing angles and coverage of the finish line and the street on that block.  The blue and green colors are the output generated by the 3D Spatial Analyst to indicate the views offered by this combination of cameras, which is created with the use of an elevation layer that indicates the location and height of visible obstructions (buildings, etc).  The camera's locations are also input with a vertical offset, as if they were being placed on the top of the adjacent buildings, which further increases the size of their field of view.  If one were attempting to assess the placement of surveillance equipment these tools would be invaluable, as they would provide a convenient means of evaluating potential camera locations by indicating specifically which areas are visible from which cameras.          

Issues of Ontological Perspective and Formalization in GIS

In the so-called “Digital Age” of the 21^st century we are privy to an ongoing debate between the merits of increased digitization of our world and the idea that this strengthening focus and reliance on computational method has the real potential to dehumanize and pervert our traditional empirical sciences.  Nadine Schuurman argues in her 2006 piece for the Annals of the Association of American Geographers “Formalization Matters: Critical GIS and Ontology Research” that this debate is unnecessarily distracting within the field of GIScience, and that the ontological focus that many feel is required is now being fully integrated within the discipline itself. 

Addressing the issues raised by many traditionalists within the GIScience field regarding the increased need for perspectives critical of the process of formalizing the conceptual, particularly with respect to those processes involving computational algorithms, needs to acknowledge the concept of “Code Space” (Schuurman, 2006, p. 729).  “Code Space” is the necessary intertwining of computer code with virtually every aspect of our day-to-day lives, including those aspects which are affected by spatial models, analysis and decision making.  Those who may be wary of this increased focus on the computational aspects of GIScience, within the discipline of geography in general, need to recognize that this shift is somewhat inevitable, given society’s prevailing and ubiquitous integration of computers and computer code/algorithms.  Schuurman argues that because this shift is common to every area of modern life, including geography’s formalized representation of the real world, it is necessary for those opposed to the paradigm shift to at least recognize the efforts being made at integrating an awareness of ontological and epistemological principles into GIScience (p. 735).

Expressing any kind of natural or human spatial relationship involving the physical environment is a process of abstraction, of formalizing the conceptual, and of necessarily losing some amount of nuance and detail.  Computer code and models exemplify this simplification, regardless of their attempts at complex accuracy, and at times it is completely necessary to purposely compromise the detailed accuracy of spatial analysis and cartographic information, in order to render it more palatable to certain audiences (p. 730).  Whether it is even possible to produce a completely accurate model or representation of these processes and analytics is a debate between Epistemelogical and Ontological Complexity, and is not likely to ever be truly definitively settled.  Schuurman contends, though, that the implicit acknowledgement and recognition of ontological principles is now becoming a dedicated facet of the disciplines of GIScience, GIS and geography (p. 735).  If the mere recognition that formalization with computational methods raise unique and fundamental ontological and epistemological issues can become innate within the formal framework of these disciplines, the traditionalists should have nothing to fear.  As long as the dialectic remains present, address and awareness of potential issues will abide.

 

Reference source:

Schuurman, N. (2006). Formalization Matters: Critical GIS and Ontology Research. Annals Of The Association Of American Geographers, 96(4), 726-739. doi:10.1111/j.1467-8306.2006.00513.x

Debugging, Error Handling, and more Trials of Perseverance & Stoicism

Programming errors are unavoidable.  For the novice programmer they are an omnipresent threat, waiting at the ready to crash a script and destroy the prospect of an early evening, to test the already frayed nerves and severely waning patience of a long suffering student.  They are how we learn- the crux of our trial and error Python education, and we must adapt to handle them with grace.  Or at least know how to deal with and fix them, and not smash our computers in the process.

Debugging script, or finding and remedying errors therein, is a skill and an art in and of itself.  There are plenty of ways to do it- the dubugging tools in PythonWin, for example, can be exceedingly helpful, or inserting print statements to isolate where in the script the error is occurring.  Finding one's preferred method is, appropriately, a trial and error process.  The screenshot above is the output of one of three simple scripts, the task at hand was to isolate and fix the errors secreted within.  Simple syntax errors- incorrect capitalization, indentation or spelling- will prevent a script from running entirely, and, like all typos, are easy to inadvertently add when typing.  These errors are relatively easy to find and fix, though, especially with the spellcheck-like "check" tool available in PythonWin.  More difficult, without question, are logic and exception errors, which occur within the code's structure, and can be exceptionally difficult to isolate and fix.  These are the errors we are faced with in our second, more insidiously villainous, script.

Here we have the screenshot of an output that is a list of the layers in a map document.  This script was a rather wonderful lesson in perseverance, as well as finding and isolating exception errors.  There were eight total errors to locate, and it was not meant to be easy.  There is something to be said, however, for the satisfaction that comes from spending upwards of 2 hours scouring tutorials and Help documents, and finally reaching that Gestalt moment when all becomes clear and the error is removed.  It is one of the more gratifying experiences one can have, to be sure.  Working with the various modules, tools and processes within the script was pretty illuminating as well- as running through it line by line with the debugger several dozen times provides ample opportunity to become familiar with them.  Exception errors are not always easy to locate and remedy, but learning to do so is an invaluable skill.

The final output, which could be contended to have the easiest "solution," was also probably the trickiest, possibly because the solution wasn't very complicated.  The idea here was to allow an error to occur within the first part (part A), using a try-except statement, so that it didn't cause a runtime error that would stop the script, and the second part (part B) could produce its output regardless.  The concept behind the try-except condition was not the easiest to grasp though, which made its correct implementation quite onerous.  The idea behind it is to instruct the program to try to execute some block of code, and in the event of an exception to produce some alternate output- like an error message- rather than stopping altogether.  A condition that allows for exceptions without crashing the entire process is a useful one to know, and the time spent on this one was, like the first two, worth the trouble and frustration for the insight it provided.   

Watershed Analysis & Stream Models

The Hawaiian Island of Kauai is a volcanic landscape that receives tremendous amounts of precipitation throughout the year, like all of the Hawaiian Islands, but is generally more forested and less developed than its fellows.  Creating an accurate model of the watersheds on the island presents a unique challenge because of the excess of rain, heavy and ultimately greatly varied amounts of forest cover, and a landscape surface composed primarily of volcanic material.  All of these factors can effect the amount of surface water present, and all contribute to the size and location of the island's many rivers and streams.

One of the aims of this week's final map was to depict the visual differences between the model-produced streams and watershed layers and the vector graphics of what actually exists.  The map above shows quite clearly that there are a myriad of discrepancies between both the stream and watershed locations produced by the ArcGIS model and those actually present.  One of the user-specified parameters involved in both models is the "accumulation threshold," or the drainage area required to produce an actual stream.  If this threshold is set too high the model output will contain too few streams, if it is set too low the output will contain too many. Upon comparison between a few generated outputs created using different specified thresholds with both aerial photos and the existing vector layers, it becomes evident that creating a model that produces results accurate to reality is no easy feat.  The above was produced with a rather low threshold, which allowed for more streams in the output- presumably logical, given the climate and terrain of the island.  One outcome affected by this, though, is the size of the model-generated watershed. Clearly the model's output (in orange) falls quite short of what is recognized as the actual watershed boundary (the yellow).  This exercise was a good lesson in the potential challenges involved with creating an accurate spatial model, and how the choices of data input can drastically affect the model's results.

Python Fundamentals, part II

This week we continue with the theme of Python scripting basics, and throw in an introduction of conditional statements.  Python, like most programming languages, accomplishes a plethora of different tasks with conditional statements; one could contend intimate knowledge of their use is a cornerstone of any successful Python tutelage.  Another integral function of Python's ability to control different workflows is the loop structure- such as while and for statements.

This screenshot displays an output of a script that uses lists of strings and numbers, a random number module, and some loop and conditional statements.  The output displayed (at the bottom of the script) is a list of random numbers, with an "unlucky" number defined, identified within the random list, an output of the list with all instances of that number removed, and one of two statements- dependent upon whether the defined number was present in the random list or not. It is also the fruits of a few hours of what might be considered, on the part of this blog author, the ultimate test of "trial-and-error patience."  Merely explaining to someone how a given process works doesn't make for a very effective lesson.  To really teach someone a concept like writing Python script it is necessary to, at a certain point, state the required output, and omit the step-by-step instruction.  It is then up to the student to sink or swim- learn how to use the code and produce the results, or don't.  It is an effective teaching strategy, to be sure, but an unfortunate by-product (for those of us with pathetically short tempers) is vexation of a level difficult to enumerate in words.  The script is written, though, the task completed, and the lesson not one to be easily forgotten.

Corridor & Cost Path Analysis

The cost of movement is usually something thought of in terms of money, in the context of shipment or transport.  Cost can take the form of a whole host of other factors, though, when it refers to some expenditure to move across a landscape.  We can express cost in terms of time, energy required, or even impact on the environment.  Anything that effects movement can be modeled as a cost, which can also be translated into a predictive measure, as in the case of corridor modeling and analysis.

  
The above is a map of the results of a corridor model for black bear movement between two disconnected portions of a National Forest.  The placement of the corridor was chosen in terms of an analysis of the elevation, land cover, and distance from roads- with its final size and position determined by the preferred habitat of the black bear.  Each cost, or factor affecting the bears' possible movement, was given a weight, and the GIS output was a predictive corridor between the two protected areas.  Possible movement of something as unpredictable as a wild animal can never be definitively plotted, but expanding a possible path into a corridor, with placement based on measurable factors, can provide a useful predictive tool.

Python Fundamentals, part I

This week we begin with the fundamentals of Python- how it is structured, and how it handles various types of data, like text, numbers, and lists.  Programming language, like communicative language, uses syntax, and the way various elements included are ordered and input makes the difference between a desired output and an angry red error message.

The output pictured above- my last name and the number 36- doesn't seem like much to produce, but the script used to create it took no small amount of effort.  It was not a simple "print" (to the screen) command to return those values, but rather a structured script that took my full name, put my first, middle and last names into a list, extracted my last name from the list, took the number of letters in my last name, and finally output that number times 3.  This process is completed in with Python script, using variable assignments, functions, methods and expressions.  It is a fairly basic procedure, and may seem a bit arbitrary, but the concepts learned and mastered therein are essential to the development of a decent working knowledge of Python.  It is upon these basic skills we build a greater understanding and mastery of the programming language.  

Land Use Suitability Modeling

If you've ever taken the time to consider, for some location, why something is there, whether it be man-mad or naturally occurring, you may be able to understand why land use suitability analyses and models are done.  The particulars of how these are typically structured and carried out is our topic for this week.

The above is a comparison of two methods of suitability analysis, both performed on raster data, and both created using a weighted overlay process.  The weighted overlay model assigns a weighted relative value to each raster cell, the difference between the two maps above being the equal weights scenario assigns the same level of importance to each of the five input values, and the alternative scenario assigns a different percent weight to each.  Each cell's value in the final output is a result of combining the relative values of the five inputs, which were raster data indicating certain criteria- such as land cover, slope, distance from rivers and roads and type of soil.  The values of the input cells for each criteria were based on a 1 - 5 scale, with higher values representing locations more desirable, such as those nearer to rivers and with certain types of soil.  By combining the five weighted values for the input (criteria) rasters, an output composed of cells indicating the relative suitability of an area, based upon the desired criteria, can be produced for a location like the study area in the maps above.  

An Introduction to Python & Computer Programming for GIS

A new semester, a new challenge- and this summer finds your humble blog author embarking on an endeavor to learn the basics of writing script for GIS with Python- ESRI's scripting language of choice for ArcGIS.  It is probably relevant to point out here that my prior experience with scripting in GIS, in its totality, involved listening to a particularly vocal co-worker complain about simple VBA operations for the version 9.2 ArcGIS we were using in those years, and I do recall him mentioning Python a few times as well.  Now (many years later) we are on ArcGIS version 10.2, and Python has been declared the victor in becoming the go-to script for automating various tasks in ArcGIS.

Python lists being open source and relatively simple/user-friendly among its advantages, and it is essential for any competent GIS professional to be at least familiar with its workings.  Even better, though, is to be at least somewhat well-versed in its use for automating geoprocessing tools, and that is the aim of this course.  We begin with learning the basics of opening the scripting and interaction windows in our Python editor- PythonWin.  From there we move on to becoming familiar with some of the basics on problem solving processes with scripts, and modeling with pseudocode and flowcharts.  Like any language, Python has its idiosyncratic ins and outs, and familiarity with some of their fundamentals is necessary as a start.

The above is a screenshot of a directory of folders that was created with a Python script.  The script was run, and the folders created, on the remote desktop that we use to access ArcGIS Desktop.  ESRI does, however, provide a free-of-charge, trial version of the software for students, good for one year.  Working on the remote desktop can be slow at times, especially on weekends when it seems that everyone in class is finishing their work, so it's often easier to use the software version run locally, without the remote connection.  The script provided for this week's exercise, that creates a directory of folders that will be used this semester, specifically references the drive used on the remote desktop connection though, and as-is must be run there to work.  Upon opening the script file in the PythonWin editor I found the specific line that does this with the simple command of "courseFolder = x:\xxx."  Running the script locally, on my laptop, I changed the location referenced (the "x"s in my example here) to the information for my personal hard drive, and ran the script there.  Lo and behold, it created the file directory (as in the screenshot above) right where I specified- on my computer.  This small triumph of successfully modifying provided code is to be savored, for the impression that these will be relatively rare this semester is rather striking.  Onward and upward we will continue though, on this exploration into the arcane and hallowed world of computer programming...