Working with a relatively new R package called "eegAnalysis" through rpy2 and getting an error for a time-series object required by the FeatureEEG function.
table_query = R_DBI.dbGetQuery(DBI_Connection, "SELECT * FROM {0}".format(a_table))
where table_query is returned from a PostgreSQL database, a small portion of the data looks like
'data.frame': 52000 obs. of 68 variables:
lfg1 : num 205 200 185 183 175 ...
lfg10: num -135.1 -124.1 -127.1 -116.1 -80.1 ...
lfg11: num -132 -109 -101 -103 -116 ...
lfg12: num -259 -246 -232 -196 -203 ...
soon there-after a time-series object is created using table_query from above
ts = R_ts.timeSeries(table_query, start = 1, end = table_query.nrow, frequency = 1)
several lines below used for the important line even further below
n_columns = ts.ncol
col_middle = n_columns / 2
if round(col_middle) != col_middle:
col_middle = col_middle + 0.5
group_1 = int(col_middle)
group_2 = int(n_columns - group_1)
classification = R_base.c(R_base.rep(1, group_1), R_base.rep(2, group_2))
the important line is here
featureEEG = R_EEGA.FeatureEEG(ts, rec_Id = ts.nrow, classes_Id = classification)
FeatureEEG is a function from the R eegAnalysis package. After processing for a few moments the
following error is returned
Traceback (most recent call last): File "D:\Projects\Kaggle\UPenn_and_Mayo_Clinic_Seizure_Detection\Scripts\py_test01.py", line 135, in featureEEG = R_EEGA.FeatureEEG(ts, rec_Id = ts.nrow, classes_Id = classification) File "C:\Python33\lib\site-packages\rpy2-2.3.8-py3.3-win32.egg\rpy2\robjects\functions.py", line 86, in call return super(SignatureTranslatedFunction, self).call(*args, **kwargs) File "C:\Python33\lib\site-packages\rpy2-2.3.8-py3.3-win32.egg\rpy2\robjects\functions.py", line 35, in call res = super(Function, self).call(*new_args, **new_kwargs) rpy2.rinterface.RRuntimeError: Error in UseMethod("months") : no applicable method for 'months' applied to an object of class "c('integer', 'numeric')"*
From what I can tell, the line creating featureEEG is processed, and it is in the processing where
the error occurs. Most of the message is understandable except for the part starting at the
RRuntimeError. Could someone explain what it is saying? Or where can I find out?
Thank You.
The RRuntimeError is reporting an error message generated by R.
Here it appears to try calling a generic months(). I am suspecting that ts in your code is not of the right type. The authors of eegAnalysis might be able to help.
Related
I am trying to solve the problem statement Jigsaw-multilingual-toxic-comment-classification but when I tried to preprocess the entire file ,I got the [Errorno 30].
I also tried to solve it by:
!cp -r /kaggle/input/jigsaw-multilingual-toxic-comment-classification /kaggle/working
Even then I am not able to solve the error.
Can someone please help me out solve this error?
The first code block defines the global variables.
!cp -r /kaggle/input/jigsaw-multilingual-toxic-comment-classification /kaggle/working
SEQUENCE_LENGTH = 128
DATA_PATH = "/kaggle/working/jigsaw-multilingual-toxic-comment-classification"
BERT_PATH = "../input/bert-multi"
BERT_PATH_SAVEDMODEL = os.path.join(BERT_PATH, "/kaggle/input/bert-multi/bert_multi_from_tfhub")
OUTPUT_PATH = "/kaggle/working"
The below code block is the block afer which I am getting the error
def preprocess_and_save_dataset(unprocessed_filename, text_label='comment_text',
seq_length=SEQUENCE_LENGTH, verbose=True):
"""Preprocess a CSV to the expected TF Dataset form for multilingual BERT,
and save the result."""
dataframe = pd.read_csv(os.path.join(DATA_PATH, unprocessed_filename),
index_col='id')
processed_filename = (unprocessed_filename.rstrip('.csv') +
"-processed-seqlen{}.csv".format(SEQUENCE_LENGTH))
pos = 0
start = time.time()
while pos < len(dataframe):
processed_df = dataframe[pos:pos + 10000].copy()
processed_df['input_word_ids'], processed_df['input_mask'], processed_df['all_segment_id'] = (
zip(*processed_df[text_label].apply(preprocess_sentence)))
if pos == 0:
processed_df.to_csv(processed_filename, index_label='id', mode='w')
else:
processed_df.to_csv(processed_filename, index_label='id', mode='a',
header=False)
if verbose:
print('Processed {} examples in {}'.format(
pos + 10000, time.time() - start))
pos += 10000
return
# Process the training dataset.
preprocess_and_save_dataset(wiki_toxic_comment_data)
I tried to move the directory to the Kaggle working folder.
I am trying to implement a custom aggregation using TFF by changing the code from this tutorial . I would like to rewrite next_fn so that all the client weights are placed at the server for further computations. As federated_collect was removed from tff-nightly, I am trying to do that using federated_aggregate.
This is what I have so far:
def accumulate(x, y):
x.append(y)
return x
def merge(x, y):
x.extend(y)
return y
#tff.federated_computation(federated_server_type, federated_dataset_type)
def next_fn(server_state, federated_dataset):
server_weights_at_client = tff.federated_broadcast(
server_state.trainable_weights)
client_deltas = tff.federated_map(
client_update_fn, (federated_dataset, server_weights_at_client))
z = []
agg_result = tff.federated_aggregate(client_deltas, z,
accumulate=tff.tf_computation(accumulate),
merge=tff.tf_computation(merge),
report=tff.tf_computation(lambda x: x))
new_weights = do_smth_with_result(agg_result)
server_state = tff.federated_map(
server_update_fn, (server_state, new_weights))
return server_state
However this results in the following Exception:
File "/home/yana/Documents/Uni/Thesis/grufedatt_try.py", line 351, in <module>
def next_fn(server_state, federated_dataset):
File "/home/yana/anaconda3/envs/fedenv/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/wrappers/computation_wrapper.py", line 494, in __call__
wrapped_func = self._strategy(
File "/home/yana/anaconda3/envs/fedenv/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/wrappers/computation_wrapper.py", line 222, in __call__
result = fn_to_wrap(*args, **kwargs)
File "/home/yana/Documents/Uni/Thesis/grufedatt_try.py", line 358, in next_fn
agg_result = tff.federated_aggregate(client_deltas, z,
File "/home/yana/anaconda3/envs/fedenv/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/federated_context/intrinsics.py", line 140, in federated_aggregate
raise TypeError(
TypeError: Expected parameter `accumulate` to be of type (<<<float32[9999,96],float32[96,1024],float32[256,1024],float32[1024],float32[256,96],float32[96]>>,<float32[9999,96],float32[96,1024],float32[256,1024],float32[1024],float32[256,96],float32[96]>> -> <<float32[9999,96],float32[96,1024],float32[256,1024],float32[1024],float32[256,96],float32[96]>>), but received (<<>,<float32[9999,96],float32[96,1024],float32[256,1024],float32[1024],float32[256,96],float32[96]>> -> <<float32[9999,96],float32[96,1024],float32[256,1024],float32[1024],float32[256,96],float32[96]>>) instead.
Try using tff.aggregators.federated_sample with max_num_samples being equal to the number of clients you have.
That should be a simple drop-in replacement for how you would previously use tff.federated_collect.
In your accumulate, the issue is that you are changing number of tensors the accumulator would contain, so you get an error when accumulating more than a single accumuland. If you would want to go this way though, for a rank-1 accumuland with k elements, you could probably do something like the following instead:
#tff.tf_computation(tff.types.TensorType(tf.float32, [None, k]),
tff.types.TensorType(tf.float32, [k]))
def accumulate(accumulator, accumuland):
return tf.concat([accumulator, tf.expand_dims(accumuland, axis=0)], axis=0)
I am trying to understand more about certain surprising results i see in implementing a tf graph .
The graph i am working with is just a forest (bunch of trees). This is just a plain forward inference graph , and nothing related to training. I am sharing the snippets for 2 implementation
code snippet 1:
with tf.name_scope("main"):
def get_tree_output(offset):
loop_vars = (offset,)
leaf_indice = tf.while_loop(cond,
body,
loop_vars,
back_prop=False,
parallel_iterations=1,
name="while_loop")
tree_score = tf.gather(score_tensor, leaf_indice, name="tree-scores")
output = tf.add(tree_score, output)
leaf_indices = tf.map_fn(get_tree_output,
tree_offsets_tensor,
dtype=INT_TYPE,
parallel_iterations=n_trees,
back_prop=False,
name="tree-scores")
tree_scores = tf.gather(score_tensor, leaf_indices, name="tree-scores")
output = tf.reduce_sum(tree_scores, name="sum-output")
output = tf.sigmoid(output, name="sigmoid-output")
code snippet 2:
with tf.name_scope("main"):
tree_offsets_tensor = tf.constant(tree_offsets, dtype=INT_TYPE, name="tree_offsets_tensor")
loop_vars = (tree_offsets_tensor,)
leaf_indices = tf.while_loop(cond,
body,
loop_vars,
back_prop=False,
parallel_iterations=n_trees,
name="while_loop")
tree_scores = tf.gather(score_tensor, leaf_indices, name="tree-scores")
output = tf.reduce_sum(tree_scores, name="sum-output")
output = tf.sigmoid(output, name="sigmoid-output")
The rest of the code is exactly the same : the constant tensors , variables, condition and body for the while loop. thread and parallelism was also the same in both case
code snippet2 : takes about 500 micro sec to do inference
code snippet 1 : take about 12 milli sec to do inference
The difference is that in snippet 1 , I use map_fn to operate on tree_offset_tensor, where as in snippet 2 , I get rid of that map_fn, and just directly use that tensor, so as I understand in snippet1 get_tree_output method gets called with one element from tree_offset_tensor, we are having multiple while_loop for each individual offset value, whereas in snippet 2 we just have one while_loop that just takes multiple offset values (basically the offset_tensor).
I also tried another variation for snippet , instead of using the map_fn I write a hand written for loop
code snippet 1 (variation for loop) :
output = 0
with tf.name_scope("main"):
for offset in tree_offsets:
loop_vars = (offset,)
leaf_indice = tf.while_loop(cond,
body,
loop_vars,
back_prop=False,
parallel_iterations=1,
name="while_loop")
tree_score = tf.gather(score_tensor, leaf_indice, name="tree-scores")
output = tf.add(tree_score, output)
#leaf_indices = tf.map_fn(get_tree_output,
# tree_offsets_tensor, dtype=INT_TYPE,
# parallel_iterations=n_trees, back_prop=False,
# name="tree-scores")
#tree_scores = tf.gather(score_tensor, leaf_indices, name="tree-scores")
#output = tf.reduce_sum(tree_scores, name="sum-output")
output = tf.sigmoid(output, name="sigmoid-output")
This gives minor improvement : 9 millisec
Im trying to make a neural network. I have followed the video from
https://www.youtube.com/watch?v=S75EdAcXHKk
I have loaded the adult.data training set.
I am now on my way of training and i have these lines where the code fails.
while(epocs<5):
i = 0
for start, end in zip(range(0, len(trX), 128), range(128, len(trX), 128)):
print(trX.shape)
tr = trX[start:end]
print(tr.shape[0])
print(tr.shape[1])
self.cost = train(tr.reshape(tr.shape[0],tr.shape[1]), trY[start:end])
epocs+=1
I am strugling with an error message which is:
n.training()
File "C:\Users\Bjornars\PycharmProjects\cogs-118a\Project\NN\Network.py", line 101, in training
self.cost = train(tr.reshape(128,106), trY[start:end])
File "C:\Anaconda3\lib\site-packages\theano\compile\function_module.py", line 513, in call
allow_downcast=s.allow_downcast)
File "C:\Anaconda3\lib\site-packages\theano\tensor\type.py", line 169, in filter
data.shape))
TypeError: ('Bad input argument to theano function with name "C:\Users\Bjornars\PycharmProjects\cogs-118a\Project\NN\Network.py:84" at index 1(0-based)', 'Wrong number of dimensions: expected 2, got 1 with shape (128,).')
The shape of the array im sending in is (5000,106)
---Solved----
Used this, it expected array not number in trY
def preprocess(self,trDmatrix,labels):
for i in range(len(trDmatrix)):
numbers = [0.0]*2
numbers[int(labels[i])]= 1.0
labels[i] = numbers
return trDmatrix, labels
As part of debugging a problem that might be related to my UIVIews, I want to write a python script to run from LLDB. I had thought to extract all settings for a view in a breakpoint and all view children, to allow me to compare states. I checked out the WWDC video on the topic and then spent time reading things at lldb.llvm.org/scripting.html, and didn't find them very helpful. A web search for examples led to nothing substantially different from those.
My problem is that I'm trying to figure out how to access iOS variables at my breakpoint. The examples I've seen do things like convert numbers and mimic shell commands. Interesting stuff but not useful for my purposes. I've been reading my way through the help info with "script help(lldb.SBValue)" and the like, but it is slow going as the results are huge and it is not clear what the use patterns are. I feel like one decent example of how to traverse a few iOS objects would help me understand the system. Does anyone know of one or can share a snippet of code?
UPDATE:
I wrote this to help me track down a bug in my UIView use. I want to do a bit more work to refine this to see if I could show the whole view tree, but this was sufficient to solve my problem, so I'll put it here to save others some time.
import lldb
max_depth = 6
filters = {'_view':'UIView *', '_layer':'CALayer *', '_viewFlags':'struct'}
def print_value(var, depth, prefix):
""" print values and recurse """
global max_depth
local_depth = max_depth - depth
pad = ' ' * local_depth
name = var.GetName()
typ = str(var.GetType()).split('\n')[0].split('{')[0].split(':')[0].strip()
found = name in filters.keys() # only visit filter items children
if found:
found = (filters.get(name) == typ)
value = var.GetValue()
if value is None or str(value) == '0x00000000':
value = ''
else:
value = ' Val: %s' % value
if var.GetNumChildren() == 0 and var.IsInScope():
path = lldb.SBStream()
var.GetExpressionPath(path)
path = ' pathData: %s' % path.GetData()
else:
path = ''
print '^' * local_depth, prefix, ' Adr:', var.GetAddress(), ' Name:', name, ' Type:', typ, value, path
if var.GetNumChildren() > 0:
if local_depth < 2 or found:
print pad, var.GetNumChildren(), 'children, to depth', local_depth + 1
counter = 0
for subvar in var:
subprefix = '%d/%d' % (counter, var.GetNumChildren())
print_value(subvar, depth - 1, subprefix)
counter += 1
def printvh (debugger, command_line, result, dict):
""" print view hierarchy """
global max_depth
args = command_line.split()
if len(args) > 0:
var = lldb.frame.FindVariable(args[0])
depth = max_depth
if len(args) > 1:
depth = int(args[1])
max_depth = depth
print_value(var, depth, 'ROOT')
else:
print 'pass a variable name and optional depth'
And I added the following to my .lldbinit :
script import os, sys
# So that files in my dir takes precedence.
script sys.path[:0] = [os.path.expanduser("~/lldbpy")]
script import views
command script add -f views.printvh printvh
so that I can just type "printvh self 3" at the LLDB prompt.
Maybe this will help. Here's an example of how to dump simple local variables when a breakpoint is hit. I'm not displaying char* arrays correctly, I'm not sure how I should get the data for these to display it like "frame variable" would display it but I'll figure that out later when I have a free minute.
struct datastore {
int val1;
int val2;
struct {
int val3;
} subdata;
char *name;
};
int main (int argc, char **argv)
{
struct datastore data = {1, 5, {3}, "a string"};
return data.val2;
}
Current executable set to 'a.out' (x86_64).
(lldb) br se -l 13
Breakpoint created: 1: file ='a.c', line = 13, locations = 1
(lldb) br comm add -s python
Enter your Python command(s). Type 'DONE' to end.
> def printvar_or_children(var):
> if var.GetNumChildren() == 0 and var.IsInScope():
> path = lldb.SBStream()
> var.GetExpressionPath(path)
> print '%s: %s' % (path.GetData(), var.GetValue())
> else:
> for subvar in var:
> printvar_or_children(subvar)
>
> print 'variables visible at breakpoint %s' % bp_loc
> for var in frame.arguments:
> printvar_or_children(var)
> for var in frame.locals:
> printvar_or_children(var)
>
> DONE
(lldb) r
variables visible at breakpoint 1.1: where = a.out`main + 51 at a.c:13, address = 0x0000000100000f33, resolved, hit count = 1
argc: 1
*(*(argv)): '/'
data.val1: 1
data.val2: 5
data.subdata.val3: 3
*(data.name): 'a'
Process 84865 stopped
* thread #1: tid = 0x1f03, 0x0000000100000f33 a.out`main + 51 at a.c:13, stop reason = breakpoint 1.1
frame #0: 0x0000000100000f33 a.out`main + 51 at a.c:13
10 int main (int argc, char **argv)
11 {
12 struct datastore data = {1, 5, {3}, "a string"};
-> 13 return data.val2;
(lldb)
Tip - for sanity's sake I worked on the python over in a side text editor and pasted it into lldb as I experimented.
If you use the frame variable command in lldb to explore your variables at a given stop location, that's the same basic way that you can access them via the SBFrame that is provided to your breakpoint python command in the 'frame' object.
Hope that helps to get you started.
Did you try looking at the python LLDB formatting templates stored in:
XCode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python/lldb/formatters/objc