Sharing Custom Tools in an ArcGIS Environment

When a custom script tool is created one measure of its intrinsic value may be its potential as something that can be shared with others.  Unsurprisingly, ArcGIS provides an easy method of doing so- with a custom script tool, created in ArcMap.  All this requires is a custom toolbox and a python script, and from there just a few modifications to the input variables and the script can be shared with others, and run from ArcMap.  

The above is a dialog box, which is for user input parameters, and must be populated before the script is run.  The parameters are set in the "Properties" of the script tool, and can be specified to a plethora of different requirements.  The text in the right pane of the dialog is customizable in the "Item Description" of the script tool, and can be modified to include specific information and instructions for not only the tool itself, but also the user input fields.  

  The output of the script is the above- as layer of random points with buffers of a specified distance.  The script can also be embedded in the actual script tool as well, so that a separate .py file isn't required if it's shared with someone else.

Custom Tools within ArcMap

ArcGIS, being the powerful program that it is, provides ample customization abilities, particularly with the creation of custom tools by way of Python scripts.  Tools are accessed by a user within the GIS Desktop interface via dialog boxes, which receive the user's input file locations, desired output location, and values for whatever parameters may be specified.  One of the handiest things about them, though, is the average user's potential to create a custom tool with a Python script, and easily format it to run within the ArcMap interface- potentially by GIS users of any and all skill levels.

A custom dialog window, such as this, looks fairly identical to the system provided tools within ArcGIS, but the process behind the user interface is a custom Python script.  One only need create a new toolbox in ArcMap, and go through the steps within the Add Script Wizard to specify the script's location, and the parameters required to run the tool.  After a few minor edits within the script itself- mainly replacing lines specifying file paths with GetParameter functions- the script can be run with any valid user-specified parameters.  

  
The Results window, which opens upon the script's execution and displays messages about the success or failure of the script, can be customized from within the source-script as well.  Print statements, used in the Python environment to print messages to the Interactive Window, are changed to AddMessage functions, which add output information to the Results window.  The above are the output messages from the custom script, modified to become a custom tool within ArcMap.  The messages specified to display upon successful completion are displayed here, and are replaced with the appropriate red-lettered warnings and notifications if there are any errors or exceptions encountered instead.    

Analyzing Statistical "Hotspots" with GIS

Using GIS to identify and locate “hotspots,” or areas with statistically significant amounts of various phenomena, is both intuitive and useful, given its powerful ability to both analyze and visually display this kind of information.  Martino, et al. (2014) use GIS to this end to explore locational patterns and statistical algorithms employed in the analysis of brain cancer in “Spatio-temporalhotspots and application on a disease analysis case via GIS.”  With this research the authors sought to identify locations with statistically significant proportions of brain cancer incidence in New Mexico, with further consideration of temporal factors, for the years 1973 to 1991.  The data used for the research was a collection of 5,000 point locations of cancer incidence for the indicated time period, with the enumeration units as the counties that compose the state of New Mexico.  The authors’ examination focused on a comparison of two methods of cluster analysis- the Extended Fuzzy C Means (EFCM) and the Extended Gustafson-Kessel (EGK) algorithms- employed within the GIS environment.  The EFCM algorithm they used employs a circular shaped hotspot area, determined by the patterns of point event clusters across space, and makes a temporal comparison for a specified period of time.  The EGK method is an analogous procedure which produced very similar results, but employs an ellipsoid shaped hotspot area, and uses a different variation of membership calculation within one if its formulas.  Analysis of the results of this research lead the authors to conclude that both algorithms, used within a GIS, are equally effective in predicting the spatial area of the diffusion of this disease, and may be of use in larger-scale studies seeking to identify temporal and/or spatial determinants within a specified study area.        

Reference

Martino, F., Sessa, S., Barillari, U. S., & Barillari, M. R. (2014). Spatio-temporal hotspots and application on a disease analysis case via GIS. Soft Computing - A Fusion Of Foundations, Methodologies And Applications, (12), 2377.

http://dl.acm.org/citation.cfm?id=2689536

Working with Rasters in Python

The corollary to working with feature geometry in vector datasets could be described as the use of the Spatial Analyst extension to access the arcpy.sa module, which allows one to work with raster datasets in Python.  This module comes complete with its own tools, functions and idiosyncrasies, which were the focus of this week's Python script.

     
Reclassification of raster data can be a cumbersome process using the Spatial Analyst tools available in ArcMap, particularly when the required result is a classification raster involving the combination of several re-classified inputs.  This task is relatively simple by comparison within a Python script though, which takes each intermediary input as a temporary dataset, and saves only user specified results.  The Python script also completes with a few short lines of code what would otherwise be completed through the creation of multiple separate raster datasets with the tools available within the Spatial Analysis Toolbox.  The efficiency of the code's ability to reclassify, and then combine to create a final output of areas containing specified characteristics, various raster data cannot be overstated.  The above image represents an area with characteristics meeting all of a set of some specified conditions- namely that the land has a particular type of landcover, and a specified range of slope and aspect.  Two input raster datasets were employed in its creation- one with landcover types, and another with elevation.  The elevation data was used to create datasets with slope and aspect characteristic attributes, and the landcover was reclassified to contain only those areas categorized as "forested."  The resulting output is the combination of the suitable areas within each of the inputs, and represents the land within the specified areal extent that meets the landcover, slope and aspect suitability requirements.   

Geometries in Python

Geometry, within the context of GIS and Python programming, refers to the points, connecting lines and arrays that compose points, polyline and polygon features.  All of the vector data objects used within a GIS can be expressed in terms of these geometries, which can be easily (relatively speaking) accessed with Python script.

The task at hand for this topic was the creation of a script that output the above- the object ID, a sequential "vertex ID," the x, y coordinates and feature name- all taken from the attribute data of a polyline shapefile of a set of rivers in Hawaii.  The script has to iterate through each line segment and print the coordinates of all of the vertices, placing each pair on a separate line of the text file.  In this process we make use of the concept of the "search cursor" introduced last week, which allows the script to access each row (or record) within the feature class.  Also put into play is the for loop, which provides the means of iteration through each row (in this case each river), and further through each line segment and set of point vertices that compose each one.  The 25 different rivers within the feature class are represented as 25 different object IDs (the first number of the line), with each coordinate pair (vertex) given a sequential ID (the second number), the x, y pair, and finally the feature's name, taken from the name attribute field.  Reading and outputting data such as this from feature classes could be a very useful task for a script to complete, as the required steps, performed manually, would surely become very time consuming, especially if one were working with a large number of datasets.   

Using Python to Manipulate Spatial Data (and also possibly to test your ability to refrain from gratuitous use of hyperbole in writing...)

Computer programming, writing script, everything involved with learning the "language of the machines"(so to speak) isn't supposed to be easy. No one will tell you it's easy.  Going into your first class on the subject is intimidating, to say the least, but there isn't really anything to decently prepare an average intellect for the exact nature of the challenge ahead.  The minor victories along the way are as sweet as any devoted student has experienced- spending hours experimenting with this and that to persuade your script to perform the way you require, and finally getting it right.  The rush of satisfaction is unparalleled, but the dark and frustrating hours toiling to get there are enough to render helpless even the most stalwart resolve.  This week and last week were dedicated to grasping the finer points of spatial data manipulation with Python. Some of the concepts in Python introduced were a bit cryptic for a novice, but extensive experimentation with them in creating the assigned script was sufficient to reveal their basic workings, if not some of their many vagaries.

We have with the above screenshot, once again, a section from the printed output of the script that was created for the assignment.  The simple output displayed above is, like the previous assignments for the semester, the mere tip of the iceberg that was the required script.  The required outputs, some of which are displayed in the above screen capture, varied from a simple geodatabase created in a specified location, to a search cursor function that queried a feature class' table to output specific attributes of records meeting certain criteria.  Suffice it to say, getting the right output from the myriad processes the script performed was an epic test of patience for your humble blog author, and fortunately (for all involved) did not result in a frustrated burst of destructive rage.  For example- the concept of a "search cursor" is perhaps not so intuitive to the nascent programmer, and so employing its functionality became a heuristic lesson, as reading about or listening to someone describe these kinds of things isn't always sufficient to create an understanding.  The employment of some SQL functionality in Python and the necessity of "field delimiters" is another case in point from this week's assignment.  The varied plethora of capabilities that the humble list can have within a Python script is also something probably not immediately evident, but when iterated through with a process from a for loop this unassuming structure can be used to complete many a useful task.  In the end, though, after all of the frustration of learning and acquiring various skills and knowledge, the objective has been met- the concepts introduced in this series of lessons are now fully grasped, and a working understanding of the material covered is the prize.  It strikes one as rendering the difficult process of getting there completely worthwhile.            
     

Geoprocessing with Python

After covering a bit of geoprocessing last week, this week we delve into the heart of the matter- geoprocessing with Python.  Geoprocessing is at the heart of the ability of GIS to solve problems with models and analysis by way of powerful computing tools, and Python could be argued to be the GIS professional's key to a more advanced use of such.  Suffice it to say, though, that moving into this territory things begin to get a bit more complicated.

Our script for the week, a screen shot of the results of which is above, performs a basic series of geoprocessing tasks, executed entirely by way of Python code.  The first operation adds x-y coordinates to a shapefile with hospital locations.  The second process is a 1000 meter buffer around the hospital points, and the third dissolves the overlapping lines of the buffers, so they become a single feature.  Each task is completed via a Python command, and file names are entered as parameters, but in order to do so a "workspace" location must be set up first.  This allows the files being manipulated to be referred to by name, and not their full file path on disc, which makes the script a bit shorter and more compact.  After each operation is completed, a message is output- indicating that it was successful, and naming the operation that was performed.  This introduction to simple geoprocessing functions within Python script is the mere tip of the iceberg of Python's capability in this realm.

Geoprocessing In ArcGIS

Geoprocessing is a cornerstone of ArcGIS, and is a general term that describes most of the spatial analytic functions of the program.  In my previous jobs working with GIS I didn't really use these tools, as I was primarily making maps for visual display. The use of GIS in any capacity beyond this typically requires use of geoprocessing tools.

The graphic above is a polygon representation of a basin area, with areas containing soil types designated as "not suitable for farming" removed. This was derived from polygon vector graphics of the basin and the various soil types that this area consists of, but the notable aspect of it is that the operation to remove the specific areas with the farming-unsuitable soil was performed with a Python script/model within ArcGIS.  Normally, one might use each tool within the program- clip, select and erase- one at a time, with the end result being the polygons above.  To streamline the workflow, which would be necessary for dealing with data in greater volumes, we can create a model within ArcGIS which will perform each operation in sequence when it is run, and can also be exported and modified as a Python script.  Models exported and modified as scripts have the advantage of potentially performing more complicated analyses, can work without the necessity of having ArcMap open, and can be scheduled to run at a designated time, automatically.  In my previous GIS career at a city board of public works, which administered water, sewer and electricity services, I worked with someone who created these kinds of scripts for our daily use, and they were essential to our operation.  I feel fortunate to now have the opportunity to learn more about how that might be done, and hope to possibly provide something as useful to someone's workflow someday myself.  

Issues of Ontological Perspective and Formalization in GIS

In the so-called “Digital Age” of the 21^st century we are privy to an ongoing debate between the merits of increased digitization of our world and the idea that this strengthening focus and reliance on computational method has the real potential to dehumanize and pervert our traditional empirical sciences.  Nadine Schuurman argues in her 2006 piece for the Annals of the Association of American Geographers “Formalization Matters: Critical GIS and Ontology Research” that this debate is unnecessarily distracting within the field of GIScience, and that the ontological focus that many feel is required is now being fully integrated within the discipline itself. 

Addressing the issues raised by many traditionalists within the GIScience field regarding the increased need for perspectives critical of the process of formalizing the conceptual, particularly with respect to those processes involving computational algorithms, needs to acknowledge the concept of “Code Space” (Schuurman, 2006, p. 729).  “Code Space” is the necessary intertwining of computer code with virtually every aspect of our day-to-day lives, including those aspects which are affected by spatial models, analysis and decision making.  Those who may be wary of this increased focus on the computational aspects of GIScience, within the discipline of geography in general, need to recognize that this shift is somewhat inevitable, given society’s prevailing and ubiquitous integration of computers and computer code/algorithms.  Schuurman argues that because this shift is common to every area of modern life, including geography’s formalized representation of the real world, it is necessary for those opposed to the paradigm shift to at least recognize the efforts being made at integrating an awareness of ontological and epistemological principles into GIScience (p. 735).

Expressing any kind of natural or human spatial relationship involving the physical environment is a process of abstraction, of formalizing the conceptual, and of necessarily losing some amount of nuance and detail.  Computer code and models exemplify this simplification, regardless of their attempts at complex accuracy, and at times it is completely necessary to purposely compromise the detailed accuracy of spatial analysis and cartographic information, in order to render it more palatable to certain audiences (p. 730).  Whether it is even possible to produce a completely accurate model or representation of these processes and analytics is a debate between Epistemelogical and Ontological Complexity, and is not likely to ever be truly definitively settled.  Schuurman contends, though, that the implicit acknowledgement and recognition of ontological principles is now becoming a dedicated facet of the disciplines of GIScience, GIS and geography (p. 735).  If the mere recognition that formalization with computational methods raise unique and fundamental ontological and epistemological issues can become innate within the formal framework of these disciplines, the traditionalists should have nothing to fear.  As long as the dialectic remains present, address and awareness of potential issues will abide.

 

Reference source:

Schuurman, N. (2006). Formalization Matters: Critical GIS and Ontology Research. Annals Of The Association Of American Geographers, 96(4), 726-739. doi:10.1111/j.1467-8306.2006.00513.x

Debugging, Error Handling, and more Trials of Perseverance & Stoicism

Programming errors are unavoidable.  For the novice programmer they are an omnipresent threat, waiting at the ready to crash a script and destroy the prospect of an early evening, to test the already frayed nerves and severely waning patience of a long suffering student.  They are how we learn- the crux of our trial and error Python education, and we must adapt to handle them with grace.  Or at least know how to deal with and fix them, and not smash our computers in the process.

Debugging script, or finding and remedying errors therein, is a skill and an art in and of itself.  There are plenty of ways to do it- the dubugging tools in PythonWin, for example, can be exceedingly helpful, or inserting print statements to isolate where in the script the error is occurring.  Finding one's preferred method is, appropriately, a trial and error process.  The screenshot above is the output of one of three simple scripts, the task at hand was to isolate and fix the errors secreted within.  Simple syntax errors- incorrect capitalization, indentation or spelling- will prevent a script from running entirely, and, like all typos, are easy to inadvertently add when typing.  These errors are relatively easy to find and fix, though, especially with the spellcheck-like "check" tool available in PythonWin.  More difficult, without question, are logic and exception errors, which occur within the code's structure, and can be exceptionally difficult to isolate and fix.  These are the errors we are faced with in our second, more insidiously villainous, script.

Here we have the screenshot of an output that is a list of the layers in a map document.  This script was a rather wonderful lesson in perseverance, as well as finding and isolating exception errors.  There were eight total errors to locate, and it was not meant to be easy.  There is something to be said, however, for the satisfaction that comes from spending upwards of 2 hours scouring tutorials and Help documents, and finally reaching that Gestalt moment when all becomes clear and the error is removed.  It is one of the more gratifying experiences one can have, to be sure.  Working with the various modules, tools and processes within the script was pretty illuminating as well- as running through it line by line with the debugger several dozen times provides ample opportunity to become familiar with them.  Exception errors are not always easy to locate and remedy, but learning to do so is an invaluable skill.

The final output, which could be contended to have the easiest "solution," was also probably the trickiest, possibly because the solution wasn't very complicated.  The idea here was to allow an error to occur within the first part (part A), using a try-except statement, so that it didn't cause a runtime error that would stop the script, and the second part (part B) could produce its output regardless.  The concept behind the try-except condition was not the easiest to grasp though, which made its correct implementation quite onerous.  The idea behind it is to instruct the program to try to execute some block of code, and in the event of an exception to produce some alternate output- like an error message- rather than stopping altogether.  A condition that allows for exceptions without crashing the entire process is a useful one to know, and the time spent on this one was, like the first two, worth the trouble and frustration for the insight it provided.   

Python Fundamentals, part II

This week we continue with the theme of Python scripting basics, and throw in an introduction of conditional statements.  Python, like most programming languages, accomplishes a plethora of different tasks with conditional statements; one could contend intimate knowledge of their use is a cornerstone of any successful Python tutelage.  Another integral function of Python's ability to control different workflows is the loop structure- such as while and for statements.

This screenshot displays an output of a script that uses lists of strings and numbers, a random number module, and some loop and conditional statements.  The output displayed (at the bottom of the script) is a list of random numbers, with an "unlucky" number defined, identified within the random list, an output of the list with all instances of that number removed, and one of two statements- dependent upon whether the defined number was present in the random list or not. It is also the fruits of a few hours of what might be considered, on the part of this blog author, the ultimate test of "trial-and-error patience."  Merely explaining to someone how a given process works doesn't make for a very effective lesson.  To really teach someone a concept like writing Python script it is necessary to, at a certain point, state the required output, and omit the step-by-step instruction.  It is then up to the student to sink or swim- learn how to use the code and produce the results, or don't.  It is an effective teaching strategy, to be sure, but an unfortunate by-product (for those of us with pathetically short tempers) is vexation of a level difficult to enumerate in words.  The script is written, though, the task completed, and the lesson not one to be easily forgotten.

Python Fundamentals, part I

This week we begin with the fundamentals of Python- how it is structured, and how it handles various types of data, like text, numbers, and lists.  Programming language, like communicative language, uses syntax, and the way various elements included are ordered and input makes the difference between a desired output and an angry red error message.

The output pictured above- my last name and the number 36- doesn't seem like much to produce, but the script used to create it took no small amount of effort.  It was not a simple "print" (to the screen) command to return those values, but rather a structured script that took my full name, put my first, middle and last names into a list, extracted my last name from the list, took the number of letters in my last name, and finally output that number times 3.  This process is completed in with Python script, using variable assignments, functions, methods and expressions.  It is a fairly basic procedure, and may seem a bit arbitrary, but the concepts learned and mastered therein are essential to the development of a decent working knowledge of Python.  It is upon these basic skills we build a greater understanding and mastery of the programming language.  

An Introduction to Python & Computer Programming for GIS

A new semester, a new challenge- and this summer finds your humble blog author embarking on an endeavor to learn the basics of writing script for GIS with Python- ESRI's scripting language of choice for ArcGIS.  It is probably relevant to point out here that my prior experience with scripting in GIS, in its totality, involved listening to a particularly vocal co-worker complain about simple VBA operations for the version 9.2 ArcGIS we were using in those years, and I do recall him mentioning Python a few times as well.  Now (many years later) we are on ArcGIS version 10.2, and Python has been declared the victor in becoming the go-to script for automating various tasks in ArcGIS.

Python lists being open source and relatively simple/user-friendly among its advantages, and it is essential for any competent GIS professional to be at least familiar with its workings.  Even better, though, is to be at least somewhat well-versed in its use for automating geoprocessing tools, and that is the aim of this course.  We begin with learning the basics of opening the scripting and interaction windows in our Python editor- PythonWin.  From there we move on to becoming familiar with some of the basics on problem solving processes with scripts, and modeling with pseudocode and flowcharts.  Like any language, Python has its idiosyncratic ins and outs, and familiarity with some of their fundamentals is necessary as a start.

The above is a screenshot of a directory of folders that was created with a Python script.  The script was run, and the folders created, on the remote desktop that we use to access ArcGIS Desktop.  ESRI does, however, provide a free-of-charge, trial version of the software for students, good for one year.  Working on the remote desktop can be slow at times, especially on weekends when it seems that everyone in class is finishing their work, so it's often easier to use the software version run locally, without the remote connection.  The script provided for this week's exercise, that creates a directory of folders that will be used this semester, specifically references the drive used on the remote desktop connection though, and as-is must be run there to work.  Upon opening the script file in the PythonWin editor I found the specific line that does this with the simple command of "courseFolder = x:\xxx."  Running the script locally, on my laptop, I changed the location referenced (the "x"s in my example here) to the information for my personal hard drive, and ran the script there.  Lo and behold, it created the file directory (as in the screenshot above) right where I specified- on my computer.  This small triumph of successfully modifying provided code is to be savored, for the impression that these will be relatively rare this semester is rather striking.  Onward and upward we will continue though, on this exploration into the arcane and hallowed world of computer programming...