Showing posts with label data manipulation. Show all posts
Showing posts with label data manipulation. Show all posts

Thursday, August 11, 2022

Roadway Index Consolidation

The fire department needs an updated wall map of city roadways with a lookup grid and an alphabetically sorted index of roadway names for rapid review. This workflow creates a grid index, and joins its identification information to all roadways within each cell; which are then consolidated into a reference index.

While this illustration uses a gridded map index, this method can be used to summarize features that exist across any kind of aerial boundary unit: for instance zip codes, districts, neighborhoods, etc.

Here is the City of Cocoa's footprint:

Create an Area of Interest Layer

This is for firefighters at Station 2, so their area of interest is mainly focused in the southeast portion of the city and surrounding areas. I created a polygon layer to describe that footprint, using ArcGIS Pro's Direction tool in the Edit toolset to snap digitized vertexes to cardinal directions for 90º corners.
ℹ Precision isn't super important here, so I didn't switch over to a Mercator projection for digitization. I left the coordinate system in NAD83 Florida State Plane East (FIPS 0901) US Feet to avoid errors in the next step which is more important.

Create Fishnet Grid

We'll test ½ square mile grid cells (2640' x 2640') to balance fitting a sufficient amount of streets into a grid cell without being too densely populated and hard to identify a roadway within a cell.
  • Open the Create Fishnet tool
  • In ArcGIS Pro 3.0, open the dropdown at the top of the Template Extent section and choose Browse to select the Area of Interest layer that we created above as the template for the extent of this grid feature class
  • Ensure you're using the appropriate coordinate system with expected unit of measure for your own needs. I'm using FL State Plane East with US Feet as the unit of measure so I can enter a value of feet for an accurate grid cell size
    • Enter 2640 in the Cell Size Width & Cell Size Height fields (which is half a mile in US feet)
    • Set the Geometry Type to Polygon and run the tool

Cull & Customize the Grid

Many of these cells are over water, and I'd prefer to get rid of those and use that space for the Index. Even if the index doesn't fit there and I end up printing it out separately, I'd still prefer to remove those unnecessary cells in the northeast. 
  • Open the Attribute Table
  • Add a Field to named Status
  • Manually select of those unwanted cells (107-110, and down running southeast to cell 33) 
  • Calculate Values of "Blank" or "Remove" or whatever
  • Open the Layer Properties for that Fishnet layer and navigate to the Definition Query section
  • Click New definition query
    • Where Status | Is Null

Re-Calculate Grid IDs

This will remove those features from the view, but the grid ID numbers will become inconsistent: e.g. After cell 32, numbering will continue on the next line above at 34.


Since the definition query is already in place, and the intended order is already sorted based on the object ID, we can run a custom field calculation to create a new contiguous unique ID to fix these gaps.
ℹ Optional: To be safe, you can create a new field (named ID_Original) to preserve the original ID. You'll first need to temporarily disable the definition query to record all values; Calculate Field on that new ID_Original and simply use the OID field to populate the ID value for every record. When finished, re-apply the definition query.
  • Create another new field named ID_Truncated or something similar
  • Ensure there is no selection, and the definition query is set to exclude any unwanted parcels (see the graphic above)
  • Open the Calculate Field dialog to calculate values on that new field
    • ID_Truncated = autoIncrement()
    • Code Block: Enter the following script and click Apply to run
rec=0
def autoIncrement():
    global rec
    pStart = 1  # adjust start value, if req'd 
    pInterval = 1  # adjust interval value, if req'd
    if (rec == 0): 
        rec = pStart 
    else: 
        rec = rec + pInterval 
    return rec


  • Finally, change the labeling field to the new ID_Truncated 

Join Grid IDs to the Roadways

Now run an Intersect process to join the grid cell IDs to each roadway, while splitting roadway segments at the borders of each grid cell
  • Input Features: the Roadway Centerlines layer, and the Fishnet grid layer
  • Attributes: All
  • Output Type: Line
The result will be a line dataset set to the extent of the fishnet grid:

Export to Spreadsheet Editor

Move over to Excel/Sheets to finish processing. Run the Table to Excel tool to export the attribute table of the resulting Intersect process. This will make it easier to consolidate the dataset into distinct roadways with multiple IDs concatenated together. Here's an example:

  • Open the dataset in Excel and hide all field besides the Address Label (from the Roadway Centerlines dataset) and the ID_Truncated field (from the fishnet grid)
  • Add three field headings after these two fields: named Count, End, and Formatted

Clean up the Data

  • Select the first row of headings, navigate to the Data tab and click the Filter button to turn on field filtering tools
  • Sort the Address Label field, ascending
  • Remove any blank or null Address Labels by selecting those records and deleting those rows
  • Select cell A1 and press Ctrl+A to select all records
  • Navigate to the Data tab and run the Remove Duplicates tool
    • Select the Address Label and the ID_Truncated fields and click OK to run. The tool will report how many duplicate records were removed

Create Analysis & Formatting Formulas

Use the following formulas for those three new fields. Count and End are for reference and filtering use later, while the Formatted field is going to help build the index mapping that we're summarizing
  • Enter the following formulas for each cell:
    • Count: =COUNTIF($A$2:$A$1177,A2)
      * Where 1177 is the last row in the dataset
    • End: =IF(A2=A3, "", "End")
    • Formatted, line 2: =B2
    • Formatted, line 3: =IF(A3=A2,E2 & ", " & B3, B3)
  • Fill the formulas down the the bottom of the dataset, leaving cell B2 as a unique outlier to help start the formatting trick 

Here's what's happening in each of these new fields:
  1. The Count field compares downward, and aims to report how many times an Address Label is present across the dataset. A Lane is in the dataset twice, while A Street is in the dataset three times.
  2. The End field aims to mark the end of a group of addresses. We'll use this flag for filtering reasons next. Notice the value of End on the 2nd of two records for A Lane, and the 3rd of three records for A Street.
  3. Most importantly, we're beginning by taking the first ID in the dataset and writing that to the Formatted field. On the next row, we're looking upward to see if this next record's address is the same as the previous. If so, that means our address is found on an additional grid cell, so we're concatenating it with the previous cell, separated with a comma and a space. If the previous Address Label is different, then it's a new address and we'll pull a new ID by itself.

Consolidate with Formatting

Now we'll finish up by applying that filter.
  • Make those formulas permanent by copying those cells and Paste Special > Values
  • Select the filter menu for the End field and remove the Blank category. Click OK to apply the filter
  • The unnecessary records are merely hidden here. Hit Ctrl+A to Select All records, and Paste it into a new worksheet to make the dataset permanent. Notice the spreadsheet's row numbers are now sequential 

Stylize & Finishing

  • Remove the ID_Truncated, Count, and End fields
  • Rename the remaining fields to Roadway, and Grid
  • Optionally, you may want to use the formula =PROPER(A2) or =UPPER(A2) to change the case of the Address Label field as needed
  • Adjust the formatting of the font size and typeface to suit your map's style
At last, copy and paste the range (or segments of the dataset at a time) back into your mapping project for final cartography and layout.



Saturday, December 30, 2017

Convert Large GIFs to Video


For GIFs under 100 MB, there are plenty of free tools around like Convertio that will convert GIF animations to AVI, MPEG, etc. But I'm not talking about HD reaction memes here. I'm helping to process full on NOAA AWIPS radar graphics. The following example is 2015 x 1224 px, and 300 frames long. These GIFs range from 97 to 170 MB, so the free online tools aren't going to cut it right now.

I'll do this manually with:
  • GIMP: A free, open source alternative to Photoshop
  • A GIMP plugin called Export Layers
  • Sony Vegas, or any other video editing platform - even Windows Movie Maker

Download & Install the Plugin

  • Download and install GIMP
  • Download the Export Layers plugin (zipped)
  • Copy the export_layers folder and the export_layers.py Python script file to your home directory:

    My Plug-Ins directory is: C:\Program Files\GIMP 2\lib\gimp\2.0\plug-ins but you may need to do some sleuthing depending on your installation, version, etc. If so:
    •  Go to Edit > Preferences
    • Expand the Folders group
    • Select the Plug-Ins preferences to view the Plug-Ins folder locations
    • Copy the folder listed along with your software installation location
  • Open GIMP (or restart GIMP) and check the File menu for the new Export Layers options

Open a GIF in GIMP

File > Open will load a GIF frame by frame as separate layers, in descending order. The first frame of the animation will be at the bottom, and the last frame will be located up at the top. Conveniently, the layers are automatically named with a frame position, and a frame duration in parentheses. These are helpful in case any frames (especially the first or last frames) are longer than the rest of the frames in the animation.

Export Layers

Now that the Export Layers plug-in is installed, you can immediately export the loaded GIF layers as individual frames.
  • Create a new directory to organize these frames separately since it'll only be a temporary intermediate set of files
  • Back in gimp choose File > Export Layers
  • Now simply choose the new directory that was just created to store these exports. Accept all default settings, and click Export Layers then Export to process the exports

Load Frames in Video Editing Software

  • Open your preferred video editing software
  • Find the preferences section that handles the default duration of still frames. Set this value to match the general frame rate of the GIF. In the screenshot above, most frames are 50 ms, while only the first and last frame of the animation hang a bit longer to denote the beginning and ending of the radar loop.

    In Sony Vegas:
    • Options > Preferences
    • Editing tab
    • Set New still image length (seconds) to 0.05 seconds (50 ms)
    In Windows Movie Maker:
    • Load all individual frames
    • Select all frames (Ctrl+A)
    • Navigate to the Edit tab
    • Set the Duration to 0.05 seconds
  • In Vegas, now Import the frames which will have the duration set by default
  • Next, select the first frame and expand it or manually set the Duration value for that individual frame to match the duration listed in its file name. In this example, the first frame hangs for 700 ms (0.7 seconds)
  • Repeat this step for the last frame in the animation: 1500 ms (1.5 seconds)

Export

  • Export and share as you need

Thursday, February 18, 2016

Rapidly Send Encrypted Information via Email

Never send sensitive information through email, or text. This includes passwords, social security numbers, bank account information, maybe even sensitive medical information, etc.

Sometimes you can't chat in person or over the phone to tell someone the information they need, or maybe you just need to send it to yourself for another location. Take a moment to quickly encrypt that data before sending it.

All you really need to encrypt information is a common program called 7-zip. Download and install it from 7-zip.org.

Encrypt your data:

  • Create a new text document. A Notepad or Wordpad .txt file is the fastest, but feel free to use Word, Excel, a nice looking template saved as PDF, or anything else
  • Write your sensitive data, save the file, and close the document

  • Right click on the file, navigate to the 7-zip sub-menu and choose Add to archive

  • Enter a password, choose your desired settings, and also check Encrypt file names just for an extra level of security. Click OK to continue

A zipped file with a .7z extension will be created. If a user attempts to open it, they will be prompted for a password. Without the password, file names nor the enclosed content will be able to be viewed, so any enclosed data is safer to send through e-mail, etc.


Monday, November 30, 2015

FSU - UF Football Rivalry Stats

Rivalry week is behind us and the trash talk is still going strong. RIP in peace my news feed - and my team won.

There's a bit of chatter about the all-time series record that they are indeed winning, but I wanted to look a bit further into it.

It turns out that, they aren't just leading the stats, they are pretty damn far ahead. However, the bulk of this comes from the early days of football at Florida State - but is it ridiculous to quote statistics from the 1950s and 60's? I'd definitely say, "Yes and no."

I don't mean to be any more of a revisionist by asking this question than say a team whose only score was produced by the defense. Let's just look at some numbers.


I've shown my work in case you want to review. Holler if I missed anything - I haven't double checked it.

If we try to identify a "Current Era" to measure the teams against each other, I found that the two teams are unbelievably close competitors.

Win/Loss Records


Looking at the all-time standings, UF took an early lead, and jumped ahead in two streaks in 1968-75, and again in 1981-86. Across all time, UF is in a commanding lead, sitting 10 wins ahead of FSU with a record of 34 wins, 24 losses, 2 ties, and one Choke at Doak.

Since I was born in '83, most this sounds pretty thin - especially coming from kids who weren't even alive when the Simpsons was a well established cult classic.

Let's try to start in somewhat more recent times, like from the beginning of the Bobby Bowden era, in 1976.


Despite 4 decent runs on both sides, FSU leads the series by 3 wins since 1976.

To be somewhat fair, let's pick another circumstance to change balance this investigation. I'll try again at the beginning of the Spurrier era, starting in 1990 - at least it's closer to the birth date of most college kids now.


No change. FSU still leads the series by 3 games.

Yeah, I agree it's bullshit to discount 15 losses and a tie, but we're just examining numbers here. Also, I don't see how the argument that, "games played in the 1950s and '60s are more relevant," holds any more weight than, "current history is more relevant to reality."

Let's take at another metric to see what's going on across multiple games.

Running Score Distribution

Example: Starting at 0, if UF wins with a score of 14-7, UF is awarded 7 points, and FSU begins with -7 points.



Across all time, FSU is just dominated by UF who ran up a score of 200 additional points over FSU - mainly between the first game in 1958 and the year before Bobby Bowden arrived. Good job, our collective grandparents who probably weren't associated with either school!

Let's see what has happened in more recent history.


Looking at the Bowden era to present, the overall score between the two teams puts UF in the lead by 3 points. THREE!

How about the Spurrier era to present?


It's a goddamn tie! Since 1990, FSU and UF have scored the same amount of offsetting points.

Someone please check my work. That's weird.

Conclusion

Okay, I admit rivalry isn't rivalry without context of history. I'm confident that the next few seasons are looking fierce for my Alma mater, and I don't doubt the ability to overcome those very early defects in the coming years. However, it's fair to say both teams have had some incredibly strong runs in recent history, and these quick illustrations have a surprising degree of balance.

Sunday, November 29, 2015

Color Development: Tools and Methods

I accidentally went nuts writing about the basics of digital color, so I decided to make it a separate article.  Check that out:
Color Development: Digital Color Basics
In this article, I'll list a few tools I use for color development, and discuss how I like to brew and develop colors in graphic design and cartography environments.

Color Brewing

These are some of my favorite tools to build and tune color sets.

Browser Utilities

ColorPick Eyedropper



ColorPick Eyedropper is a Chrome extension that enables you to quickly grab a color from a Web page. There's even a rad little magnifying visualization that helps zero in on the exact pixel you need.



Desktop Utilities

ColorPic


ColorPic is a great desktop tool to help snag Hex, HSV, RGB, CMYK, values, and save your own custom groups of color. There are even sliders, and grids to use to help build or tune your colors if necessary.

ColorOracle

ColorOracle is a fantastic color deficiency simulator. It's a quick way to preview your entire monitor view for any potential issues with color patterns which may be difficult to read for those with color-vision impairments. No exporting


Sunday, November 8, 2015

Manual Key-Numbering for Concentrated Map Labeling

I'm working on a map book with a hand full of feature layers across multiple pages. Each page in the map book is displayed at individual scales, so some pages are significantly wider than others; which means that the scale at which annotation is drawn (reference scale) is important to consider.

As an example, I set up some geotagged photo locations from around town:


Pages 3 and 4 aren't too dense, so automatic labeling is acceptable:

Page 1 is another story. Some of the points are too close together to use full labels:


Key numbering would be great for this page since there are areas where double digit numbers could easily be placed, but longer titles would not fit. The user will refer to a lookup table to match the label number with descriptive information. Furthermore, key numbering results in a visually cleaner look. Here's the finalized example:



In other projects I have multiple labeling schemes active (and I have read that multiple instances of key-numbering is a nightmare) so I came up with a workflow to manually create some key-numbering for your ArcMap project with a little sorting in Excel.

Workflow


  • From the properties menu, Export your point layer's attribute table and open it in Excel/Open Office Calc/Google Sheets/etc.
    • Ensure that the Object ID field exports correctly - we will need this in a later step
    • If the Object ID field does not export properly (see below), you may need to add a new field (type = short integer) named ObjectIDs or something similar, and calculate the OBJECTID field values into this new field before your export
The OID field didn't export correctly from ArcMap. 
Object IDs were calculated into a new field (Column D)before exporting from ArcMap
  • Sort your latitude field descending (largest [northernmost] values on top, smallest [southern] values at the end)

  • Add a new field named KeyLabel to record the order of your latitude values and save them as your key-numbering value
  • Calculate =Row()-1 for each record in the dataset (data beginning on row two will have a value of 1. This will be your key numbering label. Optionally, copy these cells and paste in the same place as values to get rid of the formula and preserve the value only
  • Save the spreadsheet, or export it in a format that ArcMap can read (.xls/.txt/etc.). Add it to ArcMap and it will be visible as a standalone table
  • Right click on your point layer and choose Joins and Relates > Join

  • Point to the table we just added, and choose to join based on the OBJECTID field from the layer as well as the table we're joining data from
  • Check the Attribute Table to ensure that the join was successful

  • Right click on your point layer and choose Properties
  • Navigate to the Label tab
  • Enable Labeling, and choose KeyLabel as the Label Field
  • You may need to play around with Placement Properties a bit to get the labels to draw in the best position for your dataset. When necessary, convert your labels to annotation to manually adjust their position

  • Finally, Go back to the spreadsheet and grab the values from the KeyLabel field as well as your point's name field. There are a few ways to display tables in ArcMap as a lookup for your key-numbering in your map. I simply pasted groups of data to text elements below, but you can also add data directly from a spreadsheet, etc.

Monday, August 17, 2015

Higher Quality UTM Zones Map

Until now, an image from the Wikipedia article on the UTM coordinate system has been my go-to UTM Zone reference. It's pretty okay.



I threw together a map that has been a bit more helpful as it shows US counties and some basic hydrography -  mostly from ArcGIS Online layers, and default ArcMap Template data.


Friday, July 24, 2015

Add a Web Mapping Data Service to ArcMap

I have a hole in my current coverage of waterbodies - I need to know the name of a few streams, and the dataset of Rivers doesn't include the stream I need. I don't necessarily want to download and manage another whole dataset of flowing waters. Temporarily referencing a Web mapping service in my ArcMap project would be great.

The best database I am aware of that includes comprehensive waterbody geometry and names is the USGS's National Hydrography Dataset. Thankfully, they do indeed provide a Web mapping service!

Search for a Mapping Service

A quick Google search with the phrase nhd web map returns a direct link to the USGS National Map's map service endpoint:

http://services.nationalmap.gov/arcgis/rest/services/nhd/MapServer

  • Copy this URL from the browser for use in the next section
These Map or Feature Service URLs (referred to as "service endpoints" or simply "endpoints") aren't always so easy to find, so other projects/data services may require more digging. Look for clues in the URL itself such as the /rest/services/ directories, and for minimal styling (basic-looking fonts, colors, lists, etc.).

To link to a specific portion of the map service (for instance a specific layer or layer group within the main service layer), follow some of the links under the Layers list (see below). Copy that specific URL for use in the next step if you would like. Compare the following URL that represents the Waterbody layer which is layer position 11 in the service package:




Add a Service to ArcMap

  1. Click the Add Data button 
  2. Change the Look in category to GIS Servers

  1. Double click on Add ArcGIS Server to launch the Add ArcGIS Server wizard
  2. Accept the default value to simply Use GIS services and click Next
  3. Paste the Server URL that was copied from where you found the chosen service endpont
  4. Leave the User Name and Password blank if the server is open for public use, and click Finish

Adding the Service Layer to an ArcMap Project

  • Hit the Add Data button again if you aren't already there
  • Ensure your Look In category is pointing to GIS Servers, just as we did in the last section
  • Double click on NHD on services.nationalmap
! Note: Slow double click on this element to rename this to "National Hydrography Dataset" or anything else if you don't prefer this title

  • Double click the nhd service layer to add the dataset to your project
! Note: This is a large dataset with that's referencing a ton of layers and information. Depending on the speed of your computer, your network connection, and the size of your existing ArcMap project, it may take some time for the layer to be added to your project.


Structure of the NHD map service. Notice that scale-
dependent drawing is set for a number of layer groups.