Consider we have an image of 2000 x 2000 pixels and the pixel size is 10 x 10 meters. The pixel values are also float numbers ranging from 0.00 - 10.00. This is image A.
I would like to resize image A to a quarter of its dimensions (i.e. 1000 x 1000 pixels) with a pixel size 20 x 20 meters (image B) by spatially aggregating four neighbouring pixels in non-overlapping blocks, starting from the top-left corner of the image, while the value for each pixel in image B will be a result of their arithmetic average.
I have written the following code using several sources from stackoverflow; however for some reason that I do not understand the resulting image (image B) is not always written properly and it is not readable by any of the software that I want to process it further (i.e. ArcGIS, ENVI, ERDAS etc).
I would appreciate any help
Best regards
Dimitris
import time
import glob
import os
import gdal
import osr
import numpy as np
start_time_script = time.clock()
path_ras='C:/rasters/'
for rasterfile in glob.glob(os.path.join(path_ras,'*.tif')):
rasterfile_name=str(rasterfile[rasterfile.find('IMG'):rasterfile.find('.tif')])
print 'Processing:'+ ' ' + str(rasterfile_name)
ds = gdal.Open(rasterfile,gdal.GA_ReadOnly)
ds_xform = ds.GetGeoTransform()
print ds_xform
ds_driver = gdal.GetDriverByName('Gtiff')
srs = osr.SpatialReference()
srs.ImportFromEPSG(26716)
ds_array = ds.ReadAsArray()
sz = ds_array.itemsize
print 'This is the size of the neighbourhood:' + ' ' + str(sz)
h,w = ds_array.shape
print 'This is the size of the Array:' + ' ' + str(h) + ' ' + str(w)
bh, bw = 2,2
shape = (h/bh, w/bw, bh, bw)
print 'This is the new shape of the Array:' + ' ' + str(shape)
strides = sz*np.array([w*bh,bw,w,1])
blocks = np.lib.stride_tricks.as_strided(ds_array,shape=shape,strides=strides)
resized_array = ds_driver.Create(rasterfile_name + '_resized_to_52m.tif',shape[1],shape[0],1,gdal.GDT_Float32)
resized_array.SetGeoTransform((ds_xform[0],ds_xform[1]*2,ds_xform[2],ds_xform[3],ds_xform[4],ds_xform[5]*2))
resized_array.SetProjection(srs.ExportToWkt())
band = resized_array.GetRasterBand(1)
zero_array = np.zeros([shape[0],shape[1]],dtype=np.float32)
print 'I start calculations using neighbourhood'
start_time_blocks = time.clock()
for i in xrange(len(blocks)):
for j in xrange(len(blocks[i])):
zero_array[i][j] = np.mean(blocks[i][j])
print 'I finished calculations and I am going to write the new array'
band.WriteArray(zero_array)
end_time_blocks = time.clock() - start_time_blocks
print 'Image Processed for:' + ' ' + str(end_time_blocks) + 'seconds' + '\n'
end_time = time.clock() - start_time_script
print 'Program ran for: ' + str(end_time) + 'seconds'
import time
import glob
import os
import gdal
import osr
import numpy as np
start_time_script = time.clock()
path_ras='C:/rasters/'
for rasterfile in glob.glob(os.path.join(path_ras,'*.tif')):
rasterfile_name=str(rasterfile[rasterfile.find('IMG'):rasterfile.find('.tif')])
print 'Processing:'+ ' ' + str(rasterfile_name)
ds = gdal.Open(rasterfile,gdal.GA_ReadOnly)
ds_xform = ds.GetGeoTransform()
print ds_xform
ds_driver = gdal.GetDriverByName('Gtiff')
srs = osr.SpatialReference()
srs.ImportFromEPSG(26716)
ds_array = ds.ReadAsArray()
sz = ds_array.itemsize
print 'This is the size of the neighbourhood:' + ' ' + str(sz)
h,w = ds_array.shape
print 'This is the size of the Array:' + ' ' + str(h) + ' ' + str(w)
bh, bw = 2,2
shape = (h/bh, w/bw, bh, bw)
print 'This is the new shape of the Array:' + ' ' + str(shape)
strides = sz*np.array([w*bh,bw,w,1])
blocks = np.lib.stride_tricks.as_strided(ds_array,shape=shape,strides=strides)
resized_array = ds_driver.Create(rasterfile_name + '_resized_to_52m.tif',shape[1],shape[0],1,gdal.GDT_Float32)
resized_array.SetGeoTransform((ds_xform[0],ds_xform[1]*2,ds_xform[2],ds_xform[3],ds_xform[4],ds_xform[5]*2))
resized_array.SetProjection(srs.ExportToWkt())
band = resized_array.GetRasterBand(1)
zero_array = np.zeros([shape[0],shape[1]],dtype=np.float32)
print 'I start calculations using neighbourhood'
start_time_blocks = time.clock()
for i in xrange(len(blocks)):
for j in xrange(len(blocks[i])):
zero_array[i][j] = np.mean(blocks[i][j])
print 'I finished calculations and I am going to write the new array'
band.WriteArray(zero_array)
end_time_blocks = time.clock() - start_time_blocks
print 'Image Processed for:' + ' ' + str(end_time_blocks) + 'seconds' + '\n'
end_time = time.clock() - start_time_script
print 'Program ran for: ' + str(end_time) + 'seconds'
Related
So I've taken the Deep Learning AI course by Andrew NG on coursera.
I am currently working the last assignment in week 2.
I reached the part where I have to write the forward and backward propagation function.
I managed to write the fwd_propagate function which is fairly easy.
This is the code below :
def fwd_propagate(w,b,X,y):
m = X.shape[1]
A = sigmoid(np.dot(w.T,X)+b)
J = (-1/m)*np.sum(y * np.log(A) + (1-y) * np.log(1-A))
return J
Now I have to write the bwd_propagation function but I don't know where and how to start.
Can someone help and explain to me what I should write.
This is everything I wrote so far with the tests.
import numpy as np
import matplotlib.pyplot as plt
import h5py
import scipy
from PIL import Image
from scipy import ndimage
%matplotlib inline
def load_dataset():
train_dataset = h5py.File('C:/Users/Univ/Desktop/ML Intern/Logistic-Regression-with-a-Neural-Network-mindset-master/train_catvnoncat.h5', "r")
train_set_x_orig = np.array(train_dataset["train_set_x"][:]) # your train set features
train_set_y_orig = np.array(train_dataset["train_set_y"][:]) # your train set labels
test_dataset = h5py.File('C:/Users/Univ/Desktop/ML Intern/Logistic-Regression-with-a-Neural-Network-mindset-master/test_catvnoncat.h5', "r")
test_set_x_orig = np.array(test_dataset["test_set_x"][:]) # your test set features
test_set_y_orig = np.array(test_dataset["test_set_y"][:]) # your test set labels
classes = np.array(test_dataset["list_classes"][:]) # the list of classes
train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))
test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))
return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes
train_set_x_orig, train_set_y, test_set_x_orig, test_set_y, classes = load_dataset()
index = 25
plt.imshow(train_set_x_orig[index])
print ("y = " + str(train_set_y[:,index]) + ", it's a '" + classes[np.squeeze(train_set_y[:,index])].decode("utf-8") + "' picture.")
print(str(train_set_y.shape[1]) + " This is the amount of elements in the training set")
print(str(test_set_y.shape[1]) + " This is the amount of elements in the test set")
print(str(train_set_x_orig.shape[1]) + " This is the Num_px")
print(f"{train_set_x_orig.shape[1]} This is the Num_px")
print(train_set_x_orig.shape)
X_flatten1 = train_set_x_orig.reshape(train_set_x_orig.shape[0], -1).T
X_flatten2 = train_set_y.reshape(train_set_y.shape[0], -1).T
X_flatten3 = test_set_x_orig.reshape(test_set_x_orig.shape[0], -1).T
X_flatten4 = test_set_y.reshape(test_set_y.shape[0], -1).T
print(X_flatten1)
print(X_flatten2)
print(X_flatten3)
print(X_flatten4)
print(X_flatten1.shape)
print(X_flatten2.shape)
print(X_flatten3.shape)
print(X_flatten4.shape)
print(" Let's standardize our date")
train_set_x = X_flatten1/256
test_set_x = X_flatten3/256
print(train_set_x)
print(test_set_x)
def sigmoid(x):
s = 1/(1+np.exp(-x))
return s
print ("sigmoid(0) = " + str(sigmoid(0)))
print ("sigmoid(9.2) = " + str(sigmoid(9.2)))
def initialize_with_zeros(dim):
shp=(dim,1)
w = np.zeros(shp)
b = 0
assert(w.shape == (dim, 1))
assert(isinstance(b, float) or isinstance(b, int))
return w,b
dim = 2
w, b = initialize_with_zeros(dim)
print ("w = " + str(w))
print ("b = " + str(b))
def fwd_propagate(w,b,X,y):
m = X.shape[1]
A = sigmoid(np.dot(w.T,X)+b)
J = (-1/m)*np.sum(y * np.log(A) + (1-y) * np.log(1-A))
return J
The next step is to calculate derivative for back propagation:
dw = 1/m*(np.dot(X, ((A-Y).T)))
db = 1/m*(np.sum(A-Y))
I want to create videos for multiple cameras stored in MySQL database. At intervals of every 1 hour the duration of video should be 59 mins and 59 secs. The videos should be then uploaded to azure blob storage.
However, every time a video is created, it is of a smaller duration (for example, 44 or 45 mins). I am not getting any errors. What am I doing wrong? any suggestion will be of great help. Thank You.
from threading import Thread
import threading
import cv2
import pymysql
import time
import datetime
import os
from azure.storage.blob import BlockBlobService, PublicAccess
def accessCamera(user_id, location_id, user_name, cursor):
fix_path = '/home/****/****/'
container_name = user_name
block_blob_service = BlockBlobService(account_name='******', account_key='*****+*****+****+/******')
try:
block_blob_service.create_container(container_name)
except:
pass
block_blob_service.set_container_acl(container_name, public_access=PublicAccess.Container)
cursor.execute('SELECT * FROM xxxxx where user_id = ' + str(user_id) + ' and location_id = ' + str(location_id) + '')
cameras = cursor.fetchall()
camera = {}
counter = 0
out = {}
hour = str(datetime.datetime.fromtimestamp(time.time()).strftime('%d-%m-%Y_%H'))
for x in cameras:
try:
os.makedirs(fix_path + x['ip_address'] + '_' + str(x['rtsp_port']) + '/videos/')
except:
pass
rtsp_stream_link = 'rtsp://' + x['username'] + ':' + x['password'] + '#' + x['ip_address'] + ':'
+ str(x['rtsp_port']) + '/stream=1'
out[counter] = cv2.VideoWriter(fix_path + x['ip_address'] + '_' + str(x['rtsp_port']) +
'/videos/' + str(hour) + '.avi',cv2.VideoWriter_fourcc(*'DIVX'), 1, (1920,1080))
camera[counter] = cv2.VideoCapture(rtsp_stream_link)
counter = counter + 1
print(" #==================================================")
print(" #-- Loaded Cameras Now Saving Video for Location " + str(location_id) + " - " +
str(datetime.datetime.fromtimestamp(time.time()).strftime('%d-%m-%Y_%H-%M-%S')))
print(" #==================================================")
framecount = 0
while True:
counter = 0
for x in cameras:
check,frame=camera[counter].read()
if check == True:
out[counter].write(frame)
else:
continue
minutes = str(datetime.datetime.fromtimestamp(time.time()).strftime('%M'))
if int(minutes) == 59 and framecount >= 3599:
out[counter].release()
block_blob_service.create_blob_from_path(container_name, fixpath + x['ip_address'] + '_'
+ str(x['rtsp_port']) + '/videos/' + hour + '.avi', x['ip_address'] + '_' + x['rtsp_port'] +
'/videos/' + hour + '.avi')
print("#==================================================")
print("#-- Video Saved - " +
str(datetime.datetime.fromtimestamp(time.time()).strftime('%d-%m-%Y_%H-%M-%S')))
print("#==================================================")
threading.Thread._stop(self)
exit(1)
counter = counter + 1
framecount = framecount + 1
#===========================================================================
# -- Main function
#===========================================================================
if __name__ == '__main__':
print("Time - " + str(datetime.datetime.fromtimestamp(time.time()).strftime('%d-%m-%Y_%H-%M-%S')))
flag = 1
while True:
if (str(datetime.datetime.fromtimestamp(time.time()).strftime('%S')) == '54' and
str(datetime.datetime.fromtimestamp(time.time()).strftime('%M')) == '59') or flag == 1:
print("#======================================================")
print("#-- Get All Users and Locations to create Threads - " +
str(datetime.datetime.fromtimestamp(time.time()).strftime('%d-%m-%Y_%H-%M-%S')))
print("#======================================================")
db = pymysql.connect('xxxxxxx', 'xxxxxx', 'password', 'xxxxxx')
cursor = db.cursor(pymysql.cursors.DictCursor)
cursor1 = db.cursor(pymysql.cursors.DictCursor)
cursor.execute('select locations.id,locations.user_id,users.username from locations inner
join users on users.id = locations.user_id')
user = cursor.fetchall()
for x in user:
thread = threading.Thread(target=accessCamera,args=
(int(x['user_id']),int(x['id']),x['username'],cursor1,))
print(" #==================================================")
print(' #-- Thread created for location ' + str(x['id']) + ' - ' +
str(datetime.datetime.fromtimestamp(time.time()).strftime('%d-%m-%Y_%H-%M-%S')))
print(" #==================================================")
thread.daemon = True
thread.start()
time.sleep(1)
flag = 0
# print(" #==================================================")
# print(" #-- Sleeping Current Thread for 1 minutes - " +
str(datetime.datetime.fromtimestamp(time.time()).strftime('%d-%m-%Y_%H-%M-%S')))
# print(" #==================================================")
I am using the HDF5DotNet with C# and I can read only full data as the attached image in the dataset.
The hdf5 file is too big, up to nearly 1.4GB, and if I load the whole array into the memory then it will be out of memory.
I would like to read all data from One columns
double[] values = new double[203572];
string m_Doc_01 = "data/sample/line";
HDFql.Execute("USE DIRECTORY " + "\"" + File_Directory + "\"");
HDFql.Execute("USE FILE " + "\"" + File_Name + "\"");
HDFql.Execute("CREATE CHUNKED(1, 203572) DATASET my_dataset_BS AS DOUBLE(2050, 203572)");
How to "m_Doc_01 ==> my_dataset_BS" Data
???
???
for (int i = 0; i < 2050; i++)
{
HDFql.Execute("SELECT FROM " + "\"" + m_Doc_01 + "\"" + "(1:::1) INTO MEMORY " + HDFql.VariableRegister(values));
}
To read the column that you have highlighted in the screenshot (i.e. column #0), you have to change the hyperslab to (please note the 0):
HDFql.Execute("SELECT FROM " + "\"" + m_Doc_01 + "\"" + "(, 0:::1) INTO MEMORY " + HDFql.VariableRegister(values));
That said, if you want to loop through the dataset and read one column at the time do the following (also better to register variable values before the loop starts and unregister it after the loop is done - this will increase performance):
number = HDFql.VariableRegister(values);
for(int i = 0; i < 2050; i++)
{
HDFql.Execute("SELECT FROM " + "\"" + m_Doc_01 + "\"" + "(, " + i + ":::1) INTO MEMORY " + number);
// do something with variable "values" (which contains the values of column #i)
}
HDFql.VariableUnregister(values);
I'm looking for a way to convert a decision tree trained using scikit sklearn into a decision table.
I would like to know how to parse the decision tree structure to find the decisions made at each step.
Then I would like ideas on how to structure this table.
Do you know a way or have a idea to do it?
Building on the other answer here. The following traverses the tree in the same way but generates a pandas dataframe as an output.
import sklearn
import pandas as pd
def tree_to_df(reg_tree, feature_names):
tree_ = reg_tree.tree_
feature_name = [
feature_names[i] if i != sklearn.tree._tree.TREE_UNDEFINED else "undefined!"
for i in tree_.feature
]
def recurse(node, row, ret):
if tree_.feature[node] != sklearn.tree._tree.TREE_UNDEFINED:
name = feature_name[node]
threshold = tree_.threshold[node]
# Add rule to row and search left branch
row[-1].append(name + " <= " + str(threshold))
recurse(tree_.children_left[node], row, ret)
# Add rule to row and search right branch
row[-1].append(name + " > " + str(threshold))
recurse(tree_.children_right[node], row, ret)
else:
# Add output rules and start a new row
label = tree_.value[node]
ret.append("return " + str(label[0][0]))
row.append([])
# Initialize
rules = [[]]
vals = []
# Call recursive function with initial values
recurse(0, rules, vals)
# Convert to table and output
df = pd.DataFrame(rules).dropna(how='all')
df['Return'] = pd.Series(values)
return df
Here is a sample code to convert a decision tree into a "python" code. You can easily adapt it to make a table.
All you need to do is create a global variable that is a table that is the size of the number of leaves times the number of features (or feature categories) and fill it recursively
def tree_to_code(tree, feature_names, classes_names):
tree_ = tree.tree_
feature_name = [
feature_names[i] if i != _tree.TREE_UNDEFINED else "undefined!"
for i in tree_.feature
]
print( "def tree(" + ", ".join(feature_names) + "):" )
def recurse(node, depth):
indent = " " * depth
if tree_.feature[node] != _tree.TREE_UNDEFINED:
name = feature_name[node]
threshold = tree_.threshold[node]
print( indent + "if " + name + " <= " + str(threshold)+ ":" )
recurse(tree_.children_left[node], depth + 1)
print( indent + "else: # if " + name + "<=" + str(threshold) )
recurse(tree_.children_right[node], depth + 1)
else:
impurity = tree.tree_.impurity[node]
dico, label = cast_value_to_dico( tree_.value[node], classes_names )
print( indent + "# impurity=" + str(impurity) + " count_max=" + str(dico[label]) )
print( indent + "return " + str(label) )
recurse(0, 1)
code snippet
from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import export_text
iris = load_iris()
X = iris['data']
y = iris['target']
decision_tree = DecisionTreeClassifier(random_state=0, max_depth=2)
decision_tree = decision_tree.fit(X, y)
r = export_text(decision_tree, feature_names=iris['feature_names'])
print(r)
listt= [r]
print(listt)
#########OUTPUT###########################
|--- petal width (cm) <= 0.80
| |--- class: 0
|--- petal width (cm) > 0.80
| |--- petal width (cm) <= 1.75
| | |--- class: 1
| |--- petal width (cm) > 1.75
| | |--- class: 2
I'm reading source code of opencv: cvProjectPoints2, cvRodrigues2.
In cvProjectPoints2, the Jacobian matrix is first got using cvRodrigues2( &_r, &matR, &_dRdr );, and then used to calculate the partial derivative of pixels w.r.t the rvec (axis-angle representation)。
if( dpdr_p )
{
double dx0dr[] =
{
X*dRdr[0] + Y*dRdr[1] + Z*dRdr[2],
X*dRdr[9] + Y*dRdr[10] + Z*dRdr[11],
X*dRdr[18] + Y*dRdr[19] + Z*dRdr[20]
};
double dy0dr[] =
{
X*dRdr[3] + Y*dRdr[4] + Z*dRdr[5],
X*dRdr[12] + Y*dRdr[13] + Z*dRdr[14],
X*dRdr[21] + Y*dRdr[22] + Z*dRdr[23]
};
double dz0dr[] =
{
X*dRdr[6] + Y*dRdr[7] + Z*dRdr[8],
X*dRdr[15] + Y*dRdr[16] + Z*dRdr[17],
X*dRdr[24] + Y*dRdr[25] + Z*dRdr[26]
};
for( j = 0; j < 3; j++ )
{
double dxdr = z*(dx0dr[j] - x*dz0dr[j]);
double dydr = z*(dy0dr[j] - y*dz0dr[j]);
double dr2dr = 2*x*dxdr + 2*y*dydr;
double dcdist_dr = k[0]*dr2dr + 2*k[1]*r2*dr2dr + 3*k[4]*r4*dr2dr;
double dicdist2_dr = -icdist2*icdist2*(k[5]*dr2dr + 2*k[6]*r2*dr2dr + 3*k[7]*r4*dr2dr);
double da1dr = 2*(x*dydr + y*dxdr);
double dmxdr = fx*(dxdr*cdist*icdist2 + x*dcdist_dr*icdist2 + x*cdist*dicdist2_dr +
k[2]*da1dr + k[3]*(dr2dr + 2*x*dxdr));
double dmydr = fy*(dydr*cdist*icdist2 + y*dcdist_dr*icdist2 + y*cdist*dicdist2_dr +
k[2]*(dr2dr + 2*y*dydr) + k[3]*da1dr);
dpdr_p[j] = dmxdr;
dpdr_p[dpdr_step+j] = dmydr;
}
dpdr_p += dpdr_step*2;
}
The shape of dRdr is 3*9, and from how the indices of dRdr is used:
X*dRdr[0] + Y*dRdr[1] + Z*dRdr[2], //-> dx0dr1
X*dRdr[9] + Y*dRdr[10] + Z*dRdr[11], //-> dx0dr2
X*dRdr[18] + Y*dRdr[19] + Z*dRdr[20] //-> dx0dr3
the Jacobian matrix seems to be:
dR1/dr1, dR2/dr1, ..., dR9/dr1,
dR1/dr2, dR2/dr2, ..., dR9/dr2,
dR1/dr3, dR2/dr3, ..., dR9/dr3,
But to my knowledge the Jacobian matrix should be of shape 9*3, since it's derivatives of R(1~9) w.r.t r(1~3):
dR1/dr1, dR1/dr2, dR1/dr3,
dR2/dr1, dR2/dr2, dR2/dr3,
...
...
dR9/dr1, dR9/dr2, dR9/dr3,
As the docs of cvRodrigues2 says:
jacobian – Optional output Jacobian matrix, 3x9 or 9x3, which is a
matrix of partial derivatives of the output array components with
respect to the input array components.
So am I misunderstanding the code & docs? Or is the code using other convention? Or is it a bug (not likely...)?
If you look up the docs:
src – Input rotation vector (3x1 or 1x3) or rotation matrix (3x3).
dst – Output rotation matrix (3x3) or rotation vector (3x1 or 1x3), respectively.
jacobian – Optional output Jacobian matrix, 3x9 or 9x3, which is a matrix of partial derivatives of the output array components with respect to the input array components.
As you see you can switch source and destination places(mathematically it will be exactly transposition), but the code does not account for it.
Therefore, indeed you've got a transposed Jacobian, because you switched first arguments places(from default places for their types). Switch them again, and you'll get normal Jacobian!