Repair Lat/Long Coordinate Errors on the Fly

Every once in a while, someone misses a step in processing raw lat/long coordinates. Sometimes data was transferred incorrectly, sometimes the data wasn't available.

Here's some data I threw together as an example - these are my favorite caving diving sites in Florida:

ID	Name	Latitude	Longitude
1	Jackson Blue	30.790475	-85.140083
2	Madison Blue	30.480518	83.244421
3	Orange Grove	30.127215	-83.13053
4	Peacock	30.122503	-83.1324
5	Little River	29.996677	-82.966328
6	Devil's Eye	29.83509	-82.69659
7	Devil's Ear	29.835297	-82.696568
8	Ginnie Spring	29.836178	-82.700148

Initial projection of data is missing a location

Something's wrong. I expected to see Madison Blue just north of that group of three points. (some of those spots are springs that are within a few hundred yards of each other.

! When projecting data, immediately zoom to that layer, or hit the Full Extent button  to ensure that all points were plotted properly.

ArcMap's Zoom to Full Extent tool

After I zoomed to the full extent of the data, I realized what must be wrong with the data (see below). Obviously there's a Longitude value that's missing its negative sign because I haven't been diving in India/Tibet before.

Zoom to full extent to check for mis-projected data.

A few quick tips:

• Locations on the same latitude, but on the other side of the world are probably missing or adding a negative sign
• Points off the coast of Nigeria/Ghana/Gabon, down under of Africa's west coast are probably inheriting a Null or erroneous value, so they're defaulting to the coordinate (0, 0).

• After zooming to the full extent of a map, use the Select Features tool to select those incorrect locations. In the Attribute Table, choose Show selected Records to take a look at the incorrect data.

Fix the Data

1. Start an Edit Session on that layer in question
2. Select the erroneous site. Mine is ID = 2, or Name = Madison Blue
3. From the Editor toolbar, choose the Sketch Properties button to open the Edit Sketch Properties window
4. Add a negative in front of the selected X value and press Enter or click Finish Sketch to apply the update

5. Save and Stop Editing

That's it! Piece of cake..

Corrected data plot

Color Development: Digital Color Basics

I intended to just write about how to create colors and suggest some cool tools I've found, but this got out of control. Check out the next article:

Color Development: Tools and Methods

There are various ways of building colors for a project or symbol. For general, rapid selection, users are often offered a basic color palette to select a color:

Default color palette from ArcMap

Color palette from Word

Even MS Paint offers a few basic colors

Notice that each of these provide an option to edit the color further.

Color Values

Colors commonly have names: like Black, White, Red, Orange, Yellow, Green, Blue, Purple, Cyan, Magenta, Olive Drab, etc.

! Specifics aside, Black, White, and shades of grays are referred to as colors here

Various environments like CSS, and HTML accept some reserved names to act as color values, however those names correspond to very specific numeric values (discussed below), and can be easily referenced on lists like this one. A color isn't "like red" to a computer. It's Red, or it's not.

In the world of coding, there's a need to describe color in a more articulate way than just names. Due to the diverse nature of how light acts (See Further Reading, at the end of this article), there are also a number of different methods that programmers and graphic artists use to build and digitally represent color. Although the various methods may describe a color value differently, they are basically just providing a group of three or more numerical values that act as measurements of various criteria that go into building a representation of color. There are four types of units you may expect to see in these groupings:

• Percentage - values from 0 to 100%
• 8-bit - Values from 0 to 255
• Hex - Values from 00 to FF
• As in 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, 11, 12, etc.
• Degrees - Values from 0 to 360º

Zero will always equal zero in any of the first three modes, but the "full" value values will be either 100%, 255, or FF. Those three are all equal values, written in different base units; in the same way that a measurement of 10 centimeters  is 100% of 3.93701 inches.

I'll get to degrees later in HSV/HSL.

Color Models

RGB

RGB Selector in ArcMap

The RGB triad stands for Red, Green, and Blue. Colors in this model are altered by adding or subtracting values in for any of the three primary color channels. As an "additive model," greater values result in brighter color instances, and lower values result in darker colors. This is based on a model of emitted light, where mixing red, green, and blue light produce the color white.

Users can adjust the sliders in this interface to control values of each of these channels between values of 0 and 255; where 0 = no value, and 255 = the full value of a given channel.

Alternatively, users can type a number they have in mind - this is particularly helpful when one wants to jump between increments of a values when searching for a color to use (e.g. changing the red value from 10, to 20, to 30, using the keyboard's number pad).

Color values are read as (#, #, #) or 

(Red value, Green value, Blue value). Since the RGB model is 24-bit (16.7 million possible color combinations), the values in each of the three channels will be some combination between 0 and 255.

As the example above illustrates, to build the color Red, tune the Red channel all the way up to 255, and leave Green and Blue channels at 0 since there should be no other color in Red but red. Done. Aside from various settings on your personal monitor, this is the standard, globally accepted, fullest possible value of Red.

Mixing two of the primary colors, Red, Green, or Blue produces
the "additive secondary" colors, Cyan, Magenta, and Yellow.
Mixing all three primary colors produces White.

The value (0, 0, 0) produces Black (the absence of light), and (255, 255, 255) produces White. Shades of gray are produced by matching all three values from (1, 1, 1) (darkest gray but not Black) to (254, 254. 254) (lightest gray but not White).

Back to our example of building the color Red, adding some value to the Green or Blue channels will yield some other color that may look reddish, but it isn't Red.

Additonally, the ArcMap interface gives you hints as to what other colors can be created from the current settings. With Red at 255, adjusting the Green channel will produces shades of red, through orange, and finally arriving at Yellow. Alternatively, adjusting the Blue channel will move through shades of reds and pinks and finally arrive at Magenta.

This is a good model to use to begin learning how to mix colors, but it's not my preferred model to use because it's tricky to imagine how to add a little red or remove a little green to target your preferred color.

CMYK

CMYK is a subtractive color model, used most often when printing is involved as an output medium. Pigments and inks absorb light (as opposed to how RGB works to emulate emitted light), so this model helps printers create the correct color with their ink print heads, as opposed to a computer monitor's LED pixels.

CMYK stands for Cyan-Magenta-Yellow-Key, where Key represents a black ink head. Black is used in printers as an efficient and controlled way to provide shadows, dark values, and black text to the printed material.

Just like mixing color with paint (where light is absorbed and the remaining wavelengths provide the observer perceived color), the CMYK model begins with the color white. Value is added to the channels, and lightness is removed. So zero values are White, and full values in all channels produce Black. Additionally, a full value in the Key channel also produces Black, regardless of the settings in the CMY channels.

Notice that the secondary colors in the RGB model produce Cyan, Magenta, and Yellow. Conversely, mixing two primary colors in the CMYK model will produce Red, Green, and Blue.

This is also not my favorite model to use to produce color, and it's generally just a pain in my ass when I'm printing large format maps.

HSV / HSL / HSVa / HSLa

HSV dialog in ArcMap

HSV selectors in Gimp, adjacent to RGB selectors
with the Hex code value, below

HSLa Selectors in Inkscape

These triads stand for Hue-Saturation-Value, and  Hue-Saturation-Lightness. The following "a" stands for Alpha, which is a metric of transparency. Sometimes this is included as a channel, sometimes it isn't. It's very important to recognize whether or not it's included. If you don't want to bother, make sure it is set to 100%, or 255, or FF.

This model is a bit tricky, so let's break it down.

Hue can be though of as the root color you're looking for on the color wheel: either a reddish, or a blueish, or a yellowish color, etc. The wheel starts at 0º, and continues through oranges, yellows, greens, blues etc, and eventually ends back at red at 360º; which also = 0º, but that's okay because both values are the same red.

A color wheel with 0° and 360° at the 12:00 position

Next, Saturation describes how muted gray (lack of color) or vibrant (fully colored) a color should be. This is usually a percentage, where 0 means no color is present and 100% represents the full colored potential of a color.

Examples of saturation gradients for several hues,
which correspond to the color wheel, above

! Note that the units can change between software packages. Inkscape uses 8-bit values (numbers 0-255) for each, HSLa channel. So a Hue value of 255 will correlate to 360° on the color wheel, and a 255 in the Saturation channel is equivalent to 100% saturation.

If you are building colors that are way off, you may need to check your units and convert when necessary.

Finally, Value and Lightness and are.. Also a bit tricky. Go ahead and read about the differences over on Wikipedia. Essentially, this component controls how light or dark a color will be.

Zero values in both HSV and HSL produce the color Black, regardless of Hue or Saturation settings.

The main difference between Value and Lightness is that the highest Value setting produces a fully saturated color (see the Saturation illustration above, and the ArcMap HSV selector below), and

the highest Lightness setting will produce a pure White color (see the Inkscape HSL selector below). A fully saturated color using HSL will have a Lightness value of 50% - or 128 (as Inkscape uses 8-bit values for each channel).

Hex

I didn't really touch on Hex, other than that it is another method of preserving a value of color. There's not much to it. It essentially is just a concatenation of the three or more channels in the fewest logical number of characters.

Short answer: Hexadecimal notation throws a couple of letters after the number 9 so each significant figure can be one of 16 possible values:

0 1 2 3 4 5 6 7 8  9  A  B  C  D  E  F

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Hex usually stands for RRGGBB, and sometimes for RRGGBBAA (for Alpha).

To find the Hex value of Red using the common six-digit RRGGBB method, first find the RGB value of Red:

  (255, 0, 0)

Then convert each value to its hex equivalent:

(FF, 00, 00)

Finally, remove the formatting, and stick the three values together:

FF0000
123456

For Inkscape and the few other examples of hex RGBa values, just throw another set of F's on the end. Here, we'll have a fully opaque value of Red:

FF0000FF

12345678

HTML & CSS

There are two ways to write hex color values in CSS and HTML.

There's the long way (again, using Red as an example):

#FF0000

And when all three RGB channels use doubled digits for their channel, you can use a 3-digit format:

#F00

Additional examples:

#112233 = #123

#00FF00 = #0F0

#333333 = #333

Further Reading

The physics of light are fascinating. I learned a lot about the subject in my various Remote Sensing courses in grad school, and in the practice of GIS and graphic design over the years. There are plenty of other great resources to use on these matters. Here are a few search terms to consider:

• Emitted vs. Reflected light
• Additive vs. Subtractive color
• Color models
• Primary color of light and pigment
• Light vs. Pigments
• Pigment vs. Dye
• Blackbody radiators
• Color theory

Add a Bing Maps Key for ArcMap

I was provided a key to use Bing Maps services in ArcGIS Desktop.  Here's how to set it up.

Enter a Bing Key

• Navigate to the bin directory where you installed ArcMap. Something like:

[Drive]:\Program Files (x86)\ArcGIS\Desktop10.2\bin

• Open the SetBingKey.exe program

• Paste your custom key into the text box and click OK

Load Bing Maps Layers Into a Project

Now you can add Bing layers in ArcMap:

• Click the Add Data dropdown and choose Add Data from ArcGIS Online… 

• Type Bing as a search term and hit Enter
• Click Add on any of the Bing Maps basemaps you want to use

* For versions other than 10.2, read more from Esri's Knowledge Base

3D Flight Data Visualization

I recently came across some delicious raw data from a flight tracking tool. Here's what it looks like in 3D!

Grab Flight Data

• Go to FlightAware.com
• Search for a flight and select it to . I found a flight from Minneapolis to New York City - a flight I'd like to be riding right now
• Click the track log & graph on the right information panel 

• Open Excel, Google Sheets, etc.
• Build the field headings separately since they don't select well with the data values from the Web page
• Select the data from the page, and copy it

Copy the data by itself - Build/copy headings separately

• Paste Special into your spreadsheet with text/values only - no formatting! If you directly paste the selection, there may be a bunch of graphic elements that migrate over too

Format & Convert

• Fill in the gaps of any altitude values that aren't available. I used the Fill Down (Ctrl+D) tool to duplicate the previous reading

• Make a new field called Linestring
• Use this formula to concatenate Longitude, Latitude, and Altitude together, separated with commas. Fill this down for the entire dataset. We'll come back to this in a moment

=C2 & "," & B2 & "," & H2

• Copy the following KML template and paste it into your favorite text editor (Notepad, Notepad++, Sublime Text, etc.)

<?xml version="1.0" encoding="UTF-8"?><br /> <kml xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom"><br /> <document><br />     <name>Flight Visualization</name><br />         <placemark><br />             <name>Flight Path</name><br />             <linestring><br />                 <coordinates><br />                     <!-- Replace this line with Lat,Long,Altitude values --><br />                 </coordinates><br />             </LineString><br />         </Placemark><br /> </Document><br /> </kml>

• Replace the comment on line 9 (<!-- Replace... -->, between the <coordinate> tags) with the data (not the field heading) of the new Linestring field
• Save this as Flight Visualization.kml, or something appropriate

Display in Google Earth

• Double click the new KML file to load in Google Earth
• Expand the new folder until you find the Flight Path. Right click on that Path layer and choose Properties

• Navigate to the Altitude Tab. Change the Altitude setting from Clamped to ground to Absolute. Make sure that Extend path to ground is checked
• On the Style, Color tab, choose a bright red Lines color. Use a Width of 1.0. Set a 30% Opacity for the Area symbol. I like the Filled + Outlined setting
• Play with these settings until you're happy. Click OK to start playing with your visualization

Masking Map Labels

I'm making a map with some Esri roadways basemap layers. Down around the the scale bar, legend, and other text elements is a dense part of a city with a lot going on with roadways. Some of those roadway labels are appearing in Data View and it looks like garbage in Layout View. I'd like to remove these.

Labels in Data View are cluttering up graphic elements in Print View

Quick solution

1. Switch to Data View (or double click your data frame to enter data view from your Layout View - I switched completely over to Data View in these screenshots)

2. From the Draw toolbar, add a rectangle graphic element (or other shape that fits your project) to cover the area you want to avoid labeling

The Draw toolbar

3. Open the element's Properties dialog, and choose No Color for the Fill and Outline. You can set the Outline Width to 0 if you want

That's it! Return to Layout View and reset your positioning. Any labels under your transparent graphic(s) will be masked.

Another method, rather than switching over to Data View, is to double click on your data frame to temporarily enter Data View from your layout view. It's a bit easier to draw your masking graphics while keeping your map elements visible.

I have cities labeled on this map, and one major label was being placed under my legend. Instead of converting those to graphics, I extended my transparent element slightly above where my legend is positioned, and ArcMap moves the otherwise hidden label well into view.

Make New Directory Script

I like to keep my projects organized by date. I  name new project folders with this format inside a "Project" folder on the root of my data drive:

YYYY-MM-DD - Project Description

It'll end up like this:

D:\Projects\2015-03-10 - Make New Directory Script\

It's an easy way to keep things consistently organized. I'll usually have some ambiguous reference directories in there that I don't want to be mixed in with regular projects (e.g. development, temp data, snippets, links, tools, logos, notes, etc.).

Shortcut Method

Another method is to add a Date Created sorting option to your file explorer window, but this may not help keep projects separated from those reference directories:

• Open Windows Explorer
• Press Alt + v to open the View menu, and choose the Details view. Sorting options will appear over the main file area of the window
• Right click on that details bar and choose More...
• Scroll down and check Date created to add it as a sorting option. Look around the Details menu and select/deselect parameters that are useful or unhelpful

Make a Formated Folder with Python

Let's get real. I like my method.

I'll also be honest: I'm not saving any real time here.  I can easily click the New Folder button and immediately start typing a folder name, but I have to look ALL the way down at the clock to verify the date, then I gotta type all 13 characters of the date prefix. I'm over it.

I threw these two tricks together in five lines using Python's os and datetime modules - And 40% of that just calls those modules!

 import os  
 import datetime  
 today = datetime.date.today()  
 newpath = today.strftime('D:\Projects\%Y-%m-%d - Description')  
 if not os.path.exists(newpath): os.makedirs(newpath)  

Line 3 prepares the today() date object
Line 4 formats the path and embeds the 4-digit year (capital %Y), etc.
Line 5 even checks to  make sure the path doesn't already exist.

Killer.

As long as you have Python installed, you can save this code as makeNewDirectory.py and run it. It should take just a fraction of a second to run.

Batch File

I was trying to run this as a .bat file, but it was kind of tricky and I got bored. Here's what I came up with.

mkdir easily makes a new directory, and this comment on StackOverflow pulls apart a %date% stamp with concatenation. I got them to work separately, but not together. Play around with underscores and let me know what you can find.