User:Chicklet16/sandbox

= Scripts =

create_fire_model.py
Location: W:\cpino\create_fire_model

This tool was originally created by Justin Kwong to create netCDF files for California fires and create CMAQ ready inputs.

The updates made were:


 * 1) Created two new ij_lookup tables (ijlookuptable_1km.csv and ijlookuptable_2km.csv) to process in the 1km and 2km domains.
 * 2) lbs_to_inlin_units_v2.py: Conversion factors for new species were added.
 * 3) pm_partition_v2.py: Different types of vegetation with their corresponding fractions were added.
 * 4) spec_list_v2.py: Species were added including their broader categories.
 * 5) speciation.py: Fractions for different species were added based on the type of burn and vegetation type decided from pm_partition_v2.py. Previously the fractions were only for wildfire burns.
 * 6) var_list_v2.py: Speciation variables were added in order to intialize the netCDF files attributes in the correct order.

create_rain_control_flag.py
Location: W:\cpino\create_rain_control_flag

This script reads a MET file, sums the rain totals for each day and GAI, and then determines if the rain total is above or below a threshold set. The MET file is expected to be in the following CSV column format: The output will result in multiple control files for each day listed in an input MET file (ideally 365 files). The output CSV column format will be as follows: The Application Control Flag is what will be decided in this script. If the rain level is greater than the threshold, the the ACF will be set to True, however if it is less than the threshold then the ACF is set to False. The Control Efficiency will also be decided based on the True/False value of the Application Control Flag and the EIC. Currently the ACF and Control Efficiency are decided by the following example:

Rain Threshold = 0.01    	     0.01" the rain threshold value in inches

If rain total per day & per GAI is greater than the threshold:

a) Control Efficiency = 75    	     For paved roads, reduces emissions by 25% (0.75 * x) when rain total is greater than 0.01"

b) Control Efficiency = 00     	    For unpaved, zeroes emissions (0.00 * x) when rain total is less than 0.01"

There is an option to run multiple MET files by using a batch file. The batch file should be in a CSV file and the first line will be skipped. Below is a an example of how it should look:

create_surrogate_subtraction.py
Location: W:\cpino\create_surrogate_subtraction

This script "subtracts" two spatial surrogates after they have been processed by the Spatial Allocator tool using our in-house script create_surrogates.py developed by John Stilley. This is done by finding the difference between each grid fraction and outputting the value to be mapped by surrogate2tile.py. The general formula is:

$$\ -\ \ =\ $$

It is best to subtract surrogates with only one variation between them. The two main examples are:


 * Different years with the SAME surrogate and domain (To show yearly differences)
 * Different surrogates with the SAME year and domain (To show differences when deciding between surrogates)

There is an option to do a batch run using a CSV file. The columns must be formatted as such:

* Note, the first column is a place holder for the surrogate ID column in the output txt file and file name.

This script also uses surrogate2tile_subtraction.py to make the difference plots (Currently only for CA). The scale ranges from +1.0 (red ) to -1.0 (blue). If the grid is red/orange then surrogate 1 has larger fractions and if the grid is blue/green then surrogate 2 has larger fractions. Below is an example of how to interpret the plots: < CA_916_4km_2030.txt > - < CA_650_4km_2030.txt >

These are both surrogates for Single Family Residences with the first being a possible update and the second being currently on hand.

Red/orange grids indicate that the updated surrogates has larger fractions in that area and Blue/green grids indicated that the updated surrogate has smaller fractions in that area than the surrogate on hand.

pems_pull_tool_*.py
Location: W:\cpino\PEMS

This tool is meant to pull highway emission aggregates from the Caltrans Performance Measurement System (PeMS). In order to run the script the user must first have a PeMS account to log in. There are multiple types of aggregates that can be run with variations in time span, quantity to be measured, and statistic type. Three types proved the most useful. Two use the same script, pems_pull_tool_time_series.py, and one uses pems_pull_tool_dow.py.


 * 1) pems_pull_tool_dow.py
 * 2) * Using this will give a "Day of Week" aggregate. The output will have the mean, maximum, and minimum Vehicle-Miles-Traveled (VMTs) fro each day of the week. The days are Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday, and Holidays.
 * 3) * The output file will have (# of counties) x (7 days of the week plus holidays) amount of entries.
 * 4) pems_pull_tool_time_series.py
 * 5) * The first option is a "daily" aggregate.
 * 6) ** The text file output includes the VMT sums for each day in a range of dates. As an example, if a range is set from May 1, 2020 to May 31, 2020, then there will be 31 daily totals for each county.
 * 7) ** (# of days in range) x (# of counties) = (# of entries in output)
 * 8) * The second option is "hourly"
 * 9) ** The hourly sums of VMT values for each day in a range of dates are outputted to a text file. As an example, if a range is set from January 1, 2019 to December 31, 2019 and all hours are selected, then there will be 24 hourly totals per day, per county. Or better represented as (# of counties) x (24 hours) x (365 days in range).
 * 10) ** This can lead to a large pull so the delay variable in the script should be set to ~ 30 seconds.
 * 11) ** (# of hours) x (# of days in range) x (# of counties) = (# of entries in output)

ModelBuilder MainIntersect Tool
Location: W:\cpino\DNB_Work\DNB_Parcel_Filter_Tool\ModelBuilderTool This is a toolbox that can be uploaded and used in ArcGIS Pro. The inputs for the tool are "CA_PARCELS_10190424" and a shapefile. Below is the step-by-step process of the tool:


 * 1) Iterate through the parcel folder for each county
 * 2) Select by attribute to pick relevant parcels
 * 3) Intersect the surrogate with the selected parcels
 * 4) Append the county shapefile to a final shapefile
 * 5) Delete the county shapefile

One suggestion for future development is to add a step that creates a template schema first rather than having the tool fail and then create one. Further information on this step is in the tutorial video which can be found on Microsoft Streams.

= Surrogates =

Mexico
List of surrogates updated:

Mexico Wilderness Mask - Development (12/31/2019)
Location: W:\cpino\mexico\WildernessMaskDevelopment_20191231 These were asked to be processed by Melissa for the possible development of a Wilderness Mask for Mexico. The Wilderness Mask is intended to act like the one already developed for California.

2014 Northern Baja California Emissions Inventory Project (12/09/2019)
Location: W:\cpino\mexico\ERG_package_20191209

This folder contains the surrogates of the deliverables from the Eastern Research Group (ERG). Below is a list of the surrogates along with their data source(s).

Mexico FOI Dataset (11/07/2018)
Location: W:\cpino\mexico\MEX_FOI_dataset_20181107

This folder contains shapefiles and surrogate2tile plots for the surrogates below.

Mexico Update (Early 2018)
Location: W:\cpino\mexico\MEX_package_2018

Updates were performed on the the California-bordering Mexico municipalities. These included population, total road miles, agriculture, forest land, and border crossings. The original Mexico surrogates are also contained in this folder and are within "old_shp". * Based on population (#10) surrogate.

DNB Filter Update 2020
Location: W:\cpino\DNB_Work\DnB_Filter_Update

* The surrogates here are still a work in progress* Some businesses in the original DNB based surrogates were not properly located. To improve the method of modeling these surrogates, a series of steps were developed:


 * 1) Pull businesses with certain SIC's (4-digit, 6-digit, or 8-digit codes)
 * 2) Geocode address to improve the lat/lon accuracy
 * 3) Run the shapefile through the MainIntersect tool on ModelBuilder.

For counties empty after the filter, it was suggested to default to the locations in that county before the filter.

Below is a summary of where the project left off:

DNB Update (9/10/2018)
Location: W:\cpino\DNB_Work\DnB_Pull_20180910.zip

Surrogates were re-pulled from Dun N Bradstreet based on relevant Standard Industrial Classification (SIC) codes. The surrogates and SIC's corresponding to them are listed below:

UC Davis
Location: W:\cpino\UCDavis

Surrogates produced by Yiting at UC Davis were processed through the Spatial Surrogate Tool and are listed here.

__NOINDEX__