mapping graduated symbols in arcpy for data with changing max value - mapping

I have a pyscript that I am working on and running into issues with the legend.
The goal of the script is to take a csv file with two columns (country, value) that I am joining to a world map based on country name. Once they are joined I turn it into a lyr file and then start to work on the mapping aspects of it.
I want to change the field being mapped to 'value' and then change the mapping to graduated symbols with 5 breaks (at 5,20,50,75,100 % of the max value in the map). The problem being that depending on what layer file is being used the max value changes. I have spent a lot of time searching for how to do this and have still come up short, there are probably some unnecessary bits to my code as well as I am now to python.
How can I get it to map based on graduated symbol based on 5 percent breaks with a changing max value?
Some of the code I have been working on below,
import os, sys, string, arcpy, arcpy.mapping, glob, arcgisscripting,time
from arcpy import env
call blank mxd file, set feature layer, Declare variables
mxd = arcpy.mapping.MapDocument(workspace/blank_4.mxd")
df = arcpy.mapping.ListDataFrames(mxd)[0]
'newname' is the layer file being used
sourceLayer = arcpy.mapping.Layer(newname)
Convert feature classes to layer objects
outLayer = str(newname)
curmap = workspace + filename + ".shp"
arcpy.MakeFeatureLayer_management(curmap, outLayer)
Save layer objects to layer files
LayerFile = str(outLayer) + ".lyr"
arcpy.SaveToLayerFile_management(outLayer, LayerFile)
Define variable for the new layer using LayerFile object
newlayer = arcpy.mapping.Layer(LayerFile)
Adds the new layer to top of TOC in the mxd doc map document
arcpy.mapping.AddLayer(df, newlayer, "TOP")
change the mapping field from country name to fert value
lyr = arcpy.mapping.ListLayers(mxd, newlayer, df)[0]
symbols = arcpy.mapping.Layer(C:\workspace\graduated_symbol2.lyr")
graduated symbols is a blank layer file where I created a legend with 5 breaks (5,20,50,75,100 %) but the range was created based on a max value of 22,000 and now when I run a file with a max value of 35,000 anything above 22,000 becomes blank.
How can I get it to purely work on % breaks, and have the 0/max range of the respective csv file be used??????
Update new layer with sybmology from the source layer
arcpy.ApplySymbologyFromLayer_management(lyr,symbols)
arcpy.mapping.UpdateLayer(df, lyr, symbols, True)
change symbology, colors, and breaks
if lyr.symbologyType == "GRADUATED_COLORS":
lyr.symbology.valueField = "value"
print lyr.symbology.numClasses #
Then I output this to a pdf, but that part is working.
Thanks!

Related

Splitting a string variable delimited list into individual binary variables in SPSS

I have a string variable created from a checkbox questions (Which of the following assets do you own?)
I am trying to create individual binary variables for each type of asset based on whether that number is present in the string list.
The syntax I am using cannot differentiate between 1 and 11.
do repeat wrd="1," ",2," ",3," ",4," ",5," ",6," ",7,"/NewVar= W3_CG_asset_TV_1 W3_CG_asset_radio_2 W3_CG_asset_payTV_3 W3_CG_asset_tel_4 W3_CG_asset_cellphone_5
W3_CG_asset_fridge_6 W3_CG_asset_freezer_7.
compute NewVar=char.index(W3_CG_HouseExpen1, wrd)>0.
end repeat.
do repeat wrd= ",8," ",9," ",10," ",11," ",12," ",13," ",14," ",15," ",16," ",17," ",18," ",19," /NewVar= W3_CG_asset_electricstove_8 W3_CG_asset_primusstove_9
W3_CG_asset_gasstove_10 W3_CG_asset_electrickettle_11 W3_CG_asset_microwave_12 W3_CG_asset_computer_13 W3_CG_asset_electricity_14 W3_CG_asset_geyser_15
W3_CG_asset_washingmachine_16 W3_CG_asset_workingvehicle_17 W3_CG_asset_bicycle_18 W3_CG_asset_donkeyhorse_19.
compute NewVar=char.index(W3_CG_HouseExpen1, wrd)>0.
end repeat.
I have tested this on SPSS 28.
make sure the column W3_CG_HouseExpen1 is string and length of it long enough to hold the data.
Then I added execute
data list list/W3_CG_HouseExpen1 (a50).
begin data
"1,2,11,12,"
"2,12,"
"1,2,"
"1,11,12,"
end data.
do repeat
wrd="1," ",2," ",3," ",4," ",5," ",6," ",7," ",8," ",9," ",10," ",11," ",12," ",13," ",14," ",15," ",16," ",17," ",18," ",19,"
/NewVar = W3_CG_asset_TV_1 W3_CG_asset_radio_2 W3_CG_asset_payTV_3 W3_CG_asset_tel_4 W3_CG_asset_cellphone_5 W3_CG_asset_fridge_6 W3_CG_asset_freezer_7
W3_CG_asset_electricstove_8 W3_CG_asset_primusstove_9 W3_CG_asset_gasstove_10 W3_CG_asset_electrickettle_11 W3_CG_asset_microwave_12 W3_CG_asset_computer_13 W3_CG_asset_electricity_14 W3_CG_asset_geyser_15
W3_CG_asset_washingmachine_16 W3_CG_asset_workingvehicle_17 W3_CG_asset_bicycle_18 W3_CG_asset_donkeyhorse_19.
compute NewVar=char.index(W3_CG_HouseExpen1, wrd)>0.
end repeat.
EXECUTE.
My suggestion is to run through this in reverse, erasing the values you've already recognized. So if you've got "11" and erased it, when you later search for "1" you won't find it in an "11".
I recreated a tiny exaple dataset to demonstrate on (EDIT-improved example):
data list list/W3_CG_HouseExpen1 (a50) .
begin data
"1,2,11,12,"
"11,12,"
"2,11,"
end data.
Now I do the whole process on a copy of the original W3_CG_HouseExpen1 variable so I can eat it away without damage to the original data:
string #temp(a50).
compute #temp=W3_CG_HouseExpen1.
do repeat wrd="12," "11," "2," "1," /NewVar= W3_12 W3_11 W3_2 W3_1.
compute NewVar=char.index(#temp, wrd)>0.
compute #temp=replace(#temp, wrd, ""). /*deleting the search string from the full string.
end repeat.
exe.

UK geojson displays blank folium map

I'm trying to use the following geojson for UK local authorities, but it displays a blank map.
<https://martinjc.github.io/UK-GeoJSON/json/eng/topo_lad.json -O UK_counties.json>
My table lists most of the local authorities in the same way. Not sure if its the key_on reference that's the only problem or if there's more to it.
My code so far is below:
threshold_scale = np.linspace(df_ctbl['Population'].min(), df_ctbl['Population'].max(), 6, dtype=int)
threshold_scale = threshold_scale.tolist() # change the numpy array to a list
threshold_scale[-1] = threshold_scale[-1] + 1 # make sure that the last value of the list is greater than the maximum immigration
# let Folium determine the scale.
londonf_map = folium.Map(location=[latitudeld, longitudeld], zoom_start=4, tiles='Mapbox Bright')
londonf_map.choropleth(
geo_data=uk_geo,
data=df_ctbl,
columns=['Local Authorityp', 'Population'],
key_on= 'properties.lad13nm',
threshold_scale=threshold_scale,
fill_color='YlOrRd',
fill_opacity=0.7,
line_opacity=0.2,
legend_name='UK Population in 2019',
reset=True
)
londonf_map
Can anyone help? Thanks
You can have a look at this answer : question about Mapbox bright. I think you have a problem with the tiles name "Mapbox Bright".

Different access methods to Pyro Paramstore give different results

I am following the Pyro introductory tutorial in forecasting, and trying to access the learned parameters after training the model, I get different results using different access methods for some of them (while getting identical results for others).
Here is the stripped-down reproducible code from the tutorial:
import torch
import pyro
import pyro.distributions as dist
from pyro.contrib.examples.bart import load_bart_od
from pyro.contrib.forecast import ForecastingModel, Forecaster
pyro.enable_validation(True)
pyro.clear_param_store()
pyro.__version__
# '1.3.1'
torch.__version__
# '1.5.0+cu101'
# import & prepare the data
dataset = load_bart_od()
T, O, D = dataset["counts"].shape
data = dataset["counts"][:T // (24 * 7) * 24 * 7].reshape(T // (24 * 7), -1).sum(-1).log()
data = data.unsqueeze(-1)
T0 = 0 # begining
T2 = data.size(-2) # end
T1 = T2 - 52 # train/test split
# define the model class
class Model1(ForecastingModel):
def model(self, zero_data, covariates):
data_dim = zero_data.size(-1)
feature_dim = covariates.size(-1)
bias = pyro.sample("bias", dist.Normal(0, 10).expand([data_dim]).to_event(1))
weight = pyro.sample("weight", dist.Normal(0, 0.1).expand([feature_dim]).to_event(1))
prediction = bias + (weight * covariates).sum(-1, keepdim=True)
assert prediction.shape[-2:] == zero_data.shape
noise_scale = pyro.sample("noise_scale", dist.LogNormal(-5, 5).expand([1]).to_event(1))
noise_dist = dist.Normal(0, noise_scale)
self.predict(noise_dist, prediction)
# fit the model
pyro.set_rng_seed(1)
pyro.clear_param_store()
time = torch.arange(float(T2)) / 365
covariates = torch.stack([time], dim=-1)
forecaster = Forecaster(Model1(), data[:T1], covariates[:T1], learning_rate=0.1)
So far so good; now, I want to inspect the learned latent parameters stored in Paramstore. Seems there are more than one ways to do this; using the get_all_param_names() method:
for name in pyro.get_param_store().get_all_param_names():
print(name, pyro.param(name).data.numpy())
I get
AutoNormal.locs.bias [14.585433]
AutoNormal.scales.bias [0.00631594]
AutoNormal.locs.weight [0.11947815]
AutoNormal.scales.weight [0.00922901]
AutoNormal.locs.noise_scale [-2.0719821]
AutoNormal.scales.noise_scale [0.03469057]
But using the named_parameters() method:
pyro.get_param_store().named_parameters()
gives the same values for the location (locs) parameters, but different values for all scales ones:
dict_items([
('AutoNormal.locs.bias', Parameter containing: tensor([14.5854], requires_grad=True)),
('AutoNormal.scales.bias', Parameter containing: tensor([-5.0647], requires_grad=True)),
('AutoNormal.locs.weight', Parameter containing: tensor([0.1195], requires_grad=True)),
('AutoNormal.scales.weight', Parameter containing: tensor([-4.6854], requires_grad=True)),
('AutoNormal.locs.noise_scale', Parameter containing: tensor([-2.0720], requires_grad=True)),
('AutoNormal.scales.noise_scale', Parameter containing: tensor([-3.3613], requires_grad=True))
])
How is this possible? According to the documentation, Paramstore is a simple key-value store; and there are only these six keys in it:
pyro.get_param_store().get_all_param_names() # .keys() method gives identical result
# result
dict_keys([
'AutoNormal.locs.bias',
'AutoNormal.scales.bias',
'AutoNormal.locs.weight',
'AutoNormal.scales.weight',
'AutoNormal.locs.noise_scale',
'AutoNormal.scales.noise_scale'])
so, there is no way that one method access one set of items and the other a different one.
Am I missing something here?
pyro.param() returns transformed parameters in this case to the positive reals for scales.
Here is the situation, as revealed in the Github thread I opened in parallel with this question...
Paramstore is no more just a simple key-value store - it also performs constraint transformations; quoting a Pyro developer from the above link:
here's some historical background. The ParamStore was originally just a key-value store. Then we added support for constrained parameters; this introduced a new layer of separation between user-facing constrained values and internal unconstrained values. We created a new dict-like user-facing interface that exposed only constrained values, but to keep backwards compatibility with old code we kept the old interface around. The two interfaces are distinguished in the source files [...] but as you observe it looks like we forgot to mark the old interface as DEPRECATED.
I guess in clarifying docs we should:
clarify that the ParamStore is no longer a simple key-value store
but also performs constraint transforms;
mark all "old" style interface methods as DEPRECATED;
remove "old" style interface usage from examples and tutorials.
As a consequence, it turns out that, while pyro.param() returns the results in the constrained (user-facing) space, the older method named_parameters() returns the unconstrained (i.e. for internal use only) values, hence the apparent discrepancy.
It's not difficult to verify indeed that the scales values returned by the two methods above are related by a logarithmic transformation:
import numpy as np
items = list(pyro.get_param_store().named_parameters()) # unconstrained space
i = 0
for name in pyro.get_param_store().keys():
if 'scales' in name:
temp = np.log(
pyro.param(name).item() # constrained space
)
print(temp, items[i][1][0].item() , np.allclose(temp, items[i][1][0].item()))
i+=1
# result:
-5.027793402915326 -5.0277934074401855 True
-4.600319371162187 -4.6003193855285645 True
-3.3920585732532835 -3.3920586109161377 True
Why does this discrepancy affect only scales parameters? That's because scales (i.e. essentially variances) are by definition constrained to be positive; that doesn't hold for locs (i.e. means), which are not constrained, hence the two representations coincide for them.
As a result of the question above, a new bullet has now been added in the Paramstore documentation, giving a relevant hint:
in general parameters are associated with both constrained and unconstrained values. for example, under the hood a parameter that is constrained to be positive is represented as an unconstrained tensor in log space.
as well as in the documentation of the named_parameters() method of the old interface:
Note that, in the event the parameter is constrained, unconstrained_value is in the unconstrained space implicitly used by the constraint.

Modify values programmatically SPSS

I have a file with more than 250 variables and more than 100 cases. Some of these variables have an error in decimal dot (20445.12 should be 2.044512).
I want to modify programatically these data, I found a possible way in a Visual Basic editor provided by SPSS (I show you a screen shot below), but I have an absolute lack of knowledge.
How can I select a range of cells in this language?
How can I store the cell once modified its data?
--- EDITED NEW DATA ----
Thank you for your fast reply.
The problem now its the number of digits that number has. For example, error data could have the following format:
Case A) 43998 (five digits) ---> 4.3998 as correct value.
Case B) 4399 (four digits) ---> 4.3990 as correct value, but parsed as 0.4399 because 0 has been removed when file was created.
Is there any way, like:
IF (NUM < 10000) THEN NUM = NUM / 1000 ELSE NUM = NUM / 10000
Or something like IF (Number_of_digits(NUM)) THEN ...
Thank you.
there's no need for VB script, go this way:
open a syntax window, paste the following code:
do repeat vr=var1 var2 var3 var4.
compute vr=vr/10000.
end repeat.
save outfile="filepath\My corrected data.sav".
exe.
Replace var1 var2 var3 var4 with the names of the actual variables you need to change. For variables that are contiguous in the file you may use var1 to var4.
Replace vr=vr/10000 with whatever mathematical calculation you would like to use to correct the data.
Replace "filepath\My corrected data.sav" with your path and file name.
WARNING: this syntax will change the data in your file. You should make sure to create a backup of your original in addition to saving the corrected data to a new file.

xtable with latex math symbols in the table

I'm trying to write this table with checkboxes using the xtable package. It would seem that the checkboxes that I've chosen are what is throwing the error. I'm just at a loss how to fix it.
library(xtable)
## I want to create a table with the names of some people in two columns
nStu = 10
## Create fake names
names = character(nStu)
for(i in 1:nStu){
names[i] = paste(LETTERS[i],rep(letters[i],5),sep='',collapse='')
}
## put check boxes behind each of the names
squares = rep('$ \\square $',nStu)
## Build the table
rosterTab = data.frame('Name'=names,'Mostly'=squares,'Sometimes'=squares, stringsAsFactors = FALSE)
## Now chop it in half and paste the halves together. (Yes, if nStu is odd, this will have to be fixed)
lTab = nStu%/%2
aTab = rosterTab[1:lTab, ]
bTab = rosterTab[(lTab+1):nStu, ]
outTab = cbind(aTab,bTab)
## Everything before this point runs fine.
outTab.tab = xtable(outTab,label=FALSE)
align(outTab.tab) = 'llcc||lcc'
print(outTab.tab, include.rownames=FALSE, sanitize.text.function = function(x){x})
The error message that I'm getting is:
Error in as.string(y) : Cannot coerce argument to a string
This error goes away if I use:
squares = rep('aaa',nStu)
Ideally, I want to get the names from a csv file (which I can do quite easily), and will use knitr to write this into a LaTeX document. (I want to do this for a bunch of input files, so automating this task seems useful to me.)
Here are some other ideas that I've considered:
A $\LaTeX$ only solution. Note, that there is one other difficulty (other than the squares not being in the input file), and that is that I would need to do some text manipulation on the strings in the input file.
Replacing the \square symbol with some other object that looks like a checkbox, that (through knitr) I can send to $\LaTeX$.

Resources