Gramex "$YAMLPATH" failing crashing - gramex

WORKING SCENARIO
When the gramex is ran with the following gramex.yaml:
schedule:
create-job-startup:
thread: true
function: >
logging.critical("hello world")
startup: "*"
The output we get, as expected, is:
DEBUG CRITICAL20-Sep 19:27:22 20-Sep 19:27:22 gramex__init__ hello world
FAILING SCENARIO
When the gramex is ran with the following gramex.yaml:
schedule:
create-job-startup:
thread: true
function: >
logging.critical("$YAMLPATH")
startup: "*"
The output we get, as expected, is:
ERROR 20-Sep 19:33:25 __init__ 9988 (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape (<unknown>, line 1)
Traceback (most recent call last):
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\services\__init__.py", line 242, in schedule
ioloop=info._main_ioloop)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\services\scheduler.py", line 31, in __init__
self.function = build_transform(schedule, vars={}, filename='schedule:%s' % name)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\transforms\transforms.py", line 183, in build_transform
tree = ast.parse(expr)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 1
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape

Use raw (r"...") string,
For example:
schedule:
create-job-startup:
thread: true
function: >
logging.critical(r"$YAMLPATH")

Related

Kivy & pyvisa - How to make an ObjectProperty to None after a GPIB connection?

I'm facing a real blocking issue with kivy and pyvisa and I'm really lost about how to find a solution to this.
In the code below, I have an ObjectProperty called 'device' who's initialized to None. I want to use it to start a GPIB connection. When this one is closed, I want to set the device property to None again.
All the code below is a simple example to try to find a solution to this problem, but I'm facing actually this issue in a real application. One of the functions of this application is to choose which equipment to use in a list, and it is impossible for me to know in advance which one will be available or not, or even if the property 'device' will be connected in GPIB or in another protocol. This is why it is important for me to reset it to None.
Here is the code:
main.py
from kivy.app import App
from kivy.uix.screenmanager import Screen
from kivy.properties import ObjectProperty, BooleanProperty, StringProperty
from pyvisa import ResourceManager
class NoneScreen(Screen):
# Properties
address = StringProperty('GPIB0::10::INSTR')
isConnected = BooleanProperty(False)
device = ObjectProperty(None)
# Connect/Disconnect very basicaly to a GPIB device
def connect(self):
if not self.isConnected:
print('Connect to', self.address)
rm = ResourceManager()
self.device = rm.open_resource(self.address)
self.isConnected = True
else:
print('Disconnect from', self.address)
self.device.close()
self.isConnected = False
# Test the state of the connection
def testCon(self):
try:
self.device.query('*IDN?')
except:
print('You are not connected')
else:
print('You are connected :)')
# Try something to make device properties None again
def noneFct(self):
self.device = None
# App
class MainApp(App):
def build(self):
ns = NoneScreen()
return ns
if __name__ == '__main__':
MainApp().run()
Main.kv
<NoneScreen>:
BoxLayout:
orientation: 'vertical'
Button:
text: 'Print'
on_release: print('self.device =', root.device)
Button:
text: 'Connect'
on_release: root.connect()
Button:
text: 'Test Connexion'
on_release: root.testCon()
Button:
text: 'Become None'
on_release: root.noneFct()
So, here is my problem : when I connect and disconnect an equipment from the GPIB, all is going well BUT when I try to make 'device' return to None after disconnection I have this error :
self.device = None
Connect to GPIB0::10::INSTR
You are connected :)
self.device = GPIBInstrument at GPIB0::10::INSTR
Disconnect from GPIB0::10::INSTR
self.device = GPIBInstrument at GPIB0::10::INSTR
You are not connected
[INFO ] [Base ] Leaving application in progress...
Traceback (most recent call last):
File "C:\Users\Frederic\Desktop\debug\main.py", line 51, in <module>
MainApp().run()
File "C:\Users\Frederic\AppData\Local\Programs\Python\Python37\lib\site-packages\kivy\app.py", line 855, in run
runTouchApp()
File "C:\Users\Frederic\AppData\Local\Programs\Python\Python37\lib\site-packages\kivy\base.py", line 504, in runTouchApp
EventLoop.window.mainloop()
File "C:\Users\Frederic\AppData\Local\Programs\Python\Python37\lib\site-packages\kivy\core\window\window_sdl2.py", line 747, in mainloop
self._mainloop()
File "C:\Users\Frederic\AppData\Local\Programs\Python\Python37\lib\site-packages\kivy\core\window\window_sdl2.py", line 479, in _mainloop
EventLoop.idle()
File "C:\Users\Frederic\AppData\Local\Programs\Python\Python37\lib\site-packages\kivy\base.py", line 342, in idle
self.dispatch_input()
File "C:\Users\Frederic\AppData\Local\Programs\Python\Python37\lib\site-packages\kivy\base.py", line 327, in dispatch_input
post_dispatch_input(*pop(0))
File "C:\Users\Frederic\AppData\Local\Programs\Python\Python37\lib\site-packages\kivy\base.py", line 293, in post_dispatch_input
wid.dispatch('on_touch_up', me)
File "kivy\_event.pyx", line 707, in kivy._event.EventDispatcher.dispatch
File "C:\Users\Frederic\AppData\Local\Programs\Python\Python37\lib\site-packages\kivy\uix\behaviors\button.py", line 179, in on_touch_up
self.dispatch('on_release')
File "kivy\_event.pyx", line 703, in kivy._event.EventDispatcher.dispatch
File "kivy\_event.pyx", line 1214, in kivy._event.EventObservers.dispatch
File "kivy\_event.pyx", line 1098, in kivy._event.EventObservers._dispatch
File "C:\Users\Frederic\AppData\Local\Programs\Python\Python37\lib\site-packages\kivy\lang\builder.py", line 64, in custom_callback
exec(__kvlang__.co_value, idmap)
File "C:\Users\Frederic\Desktop\debug\main.kv", line 20, in <module>
on_release: root.noneFct()
File "C:\Users\Frederic\Desktop\debug\main.py", line 38, in noneFct
self.device = None
File "kivy\properties.pyx", line 497, in kivy.properties.Property.__set__
File "kivy\properties.pyx", line 541, in kivy.properties.Property.set
File "kivy\properties.pyx", line 532, in kivy.properties.Property.set
File "kivy\properties.pyx", line 1001, in kivy.properties.ObjectProperty.check
File "kivy\properties.pyx", line 570, in kivy.properties.Property.check
ValueError: None is not allowed for NoneScreen.device
So, here is why it's going weird for me : if I try to do the same thing, without using kivy, in a python command line, it works !
>>> import pyvisa
>>> rm = pyvisa.ResourceManager()
>>> device = None
>>> print(device)
None
>>> device = rm.open_resource('GPIB0::10::INSTR')
>>> print(device)
GPIBInstrument at GPIB0::10::INSTR
>>> device.query('*IDN?')
'Agilent Technologies,33250A,0,2.01-1.01-1.00-03-2\n'
>>> device.close()
>>> print(device)
GPIBInstrument at GPIB0::10::INSTR
>>> device = None
>>> print(device)
None
So, I really don't understand why it doesn't work...
Does anyone have any idea on how to fix this ?
N.B : This my first post here, so I hope all is clearly explained and that I didn't make a mistake ^^ Please let me know if it is the case !
https://kivy.org/doc/stable/api-kivy.properties.html#kivy.properties.Property
None is a special case: you can set the default value of a Property to None, but you can’t set None to a property afterward. If you really want to do that, you must declare the Property with allownone=True:
class MyObject(Widget):
hello = ObjectProperty(None, allownone=True)
# then later
a = MyObject()
a.hello = 'bleh' # working
a.hello = None # working too, because allownone is True.
Change
device = ObjectProperty(None)
To
device = ObjectProperty(None, allownone=True)

How to use table.where() to filter for subfields in PyFlink?

I'm using pyflink and Flink 11.2 and I've defined my table like this:
def _create_sink_table(st_env):
# Create SINK table.
st_env.execute_sql(f"""
CREATE TABLE {"in"} (
`a` STRING,
`b` STRING,
`c` STRING,
`d` ROW(
`e` STRING,
`f` STRING
)
) WITH (
'connector' = 'kafka',
'topic' = '{SINK_TOPIC_NAME}',
'properties.bootstrap.servers' = '{BOOTSTRAP_SERVERS}',
'format' = 'json',
'json.fail-on-missing-field' = 'false',
'json.ignore-parse-errors' = 'true'
)
""")
I want to use table.where to filter for subfield:
def execute(st_env: StreamTableEnvironment):
table = st_env.from_path("in")
table = table.select("a, b, c, d")
table = table.where("d.e = 'fail'")
Everything I've tried so far resulted in an error, this is one of them:
➜ ./run.sh
Traceback (most recent call last):
File "/tmp/test.py", line 143, in <module>
main()
File "/tmp/test.py", line 139, in main
execute(st_env)
File "/tmp/test.py", line 109, in execute
table = table.where("d.e = 'fail'")
File "/opt/flink/opt/python/pyflink.zip/pyflink/table/table.py", line 140, in where
File "/opt/flink/opt/python/py4j-0.10.8.1-src.zip/py4j/java_gateway.py", line 1286, in __call__
File "/opt/flink/opt/python/pyflink.zip/pyflink/util/exceptions.py", line 147, in deco
File "/opt/flink/opt/python/py4j-0.10.8.1-src.zip/py4j/protocol.py", line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o39.where.
: org.apache.flink.table.api.ValidationException: Undefined function: e
at org.apache.flink.table.expressions.resolver.LookupCallResolver.lambda$visit$0(LookupCallResolver.java:49)
at java.util.Optional.orElseThrow(Optional.java:290)
at org.apache.flink.table.expressions.resolver.LookupCallResolver.visit(LookupCallResolver.java:49)
at org.apache.flink.table.expressions.resolver.LookupCallResolver.visit(LookupCallResolver.java:38)
at org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at org.apache.flink.table.expressions.LookupCallExpression.accept(LookupCallExpression.java:65)
at org.apache.flink.table.expressions.resolver.LookupCallResolver.lambda$resolveChildren$1(LookupCallResolver.java:64)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
It tries to execute the function e on d, instead of retrieving the subfield.
I think table.where("d.get(0) = 'fail'") maybe work

Problem to use the commit:_config of napalm, not finding the config file

Hey guys I am trying to test the commit of napalm, but it is not able to finding the cfg file, I also tried to change to "txt", but the same error.
Here my code:
import napalm
import json
driver = napalm.get_network_driver("ios")
device = driver(
hostname="10.0.0.254",
username="cisco",
password="cisco",
optional_args={"secret" : "cisco"}
)
device.open()
device.load_merge_candidate(filename="config.cfg")
device.commit_config()
device.close()
and this is the error,both files are in the same folder:
user#user-pc:~/Documents/python_files$ /usr/bin/python3
/home/user/Documents/python_files/network/config_compare.py Traceback
(most recent call last): File
"/home/user/Documents/python_files/network/config_compare.py", line
18, in
device.load_merge_candidate(filename="config.cfg") File "/home/user/.local/lib/python3.8/site-packages/napalm/ios/ios.py",
line 315, in load_merge_candidate
return_status, msg = self._load_candidate_wrapper( File "/home/user/.local/lib/python3.8/site-packages/napalm/ios/ios.py",
line 282, in _load_candidate_wrapper
(return_status, msg) = self._scp_file( File "/home/user/.local/lib/python3.8/site-packages/napalm/ios/ios.py",
line 620, in _scp_file
return self._xfer_file( File "/home/user/.local/lib/python3.8/site-packages/napalm/ios/ios.py",
line 670, in _xfer_file
with TransferClass(**kwargs) as transfer: File "/home/user/.local/lib/python3.8/site-packages/netmiko/ssh_dispatcher.py",
line 278, in FileTransfer
return FileTransferClass(*args, **kwargs) File "/home/user/.local/lib/python3.8/site-packages/netmiko/scp_handler.py",
line 80, in init
self.source_md5 = self.file_md5(source_file) File "/home/user/.local/lib/python3.8/site-packages/netmiko/scp_handler.py",
line 257, in file_md5
with open(file_name, "rb") as f: FileNotFoundError: [Errno 2] No such file or directory: 'config.c
tks for any help..
It looks like a problem with a filepath, are you sure that 'config.cfg' is in the same dir as your code?
Try the below to see if you can access the file (if you can't you need to modify the filepath):
import napalm
driver = napalm.get_network_driver("ios")
device = driver(
hostname="10.0.0.254",
username="cisco",
password="cisco",
optional_args={"secret" : "cisco"}
)
device.open()
print('This is a config to be added:\n')
with open('config.cfg','r') as f:
output = f.read()
device.load_merge_candidate(filename="config.cfg")
# I would add this to see what's you are merging:
print(device.compare_config())
confirm_config= input('Do you want to deploy the above config? Press Y to deploy \n')
if confirm_config == 'Y':
device.commit_config()
device.close()
else:
device.close()

When using building a federated averaging process - TypeError: Expected a callable.... found Enhanced Model

1 issue at large
I am producing a iterative process via tff.learning.build_federated_averaging_process(). and receive the error:
Traceback (most recent call last):
File "B:\tools and software\Anaconda\envs\bookProjects\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-2-47998fd56829>", line 1, in <module>
runfile('B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/train_v04.py', args=['--experiment_name=temp', '--client_batch_size=20', '--client_optimizer=sgd', '--client_learning_rate=0.2', '--server_optimizer=sgd', '--server_learning_rate=1.0', '--total_rounds=200', '--rounds_per_eval=1', '--rounds_per_checkpoint=50', '--rounds_per_profile=0', '--root_output_dir=B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/logs/fed_out/'], wdir='B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection')
File "B:\tools and software\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "B:\tools and software\PyCharm 2020.1\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/train_v04.py", line 306, in <module>
app.run(main)
File "B:\tools and software\Anaconda\envs\bookProjects\lib\site-packages\absl\app.py", line 299, in run
_run_main(main, args)
File "B:\tools and software\Anaconda\envs\bookProjects\lib\site-packages\absl\app.py", line 250, in _run_main
sys.exit(main(argv))
File "B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/train_v04.py", line 299, in main
train_main()
File "B:/projects/openProjects/githubprojects/BotnetTrafficAnalysisFederaedLearning/anomaly-detection/train_v04.py", line 262, in train_main
server_optimizer_fn=server_optimizer_fn,
File "B:\tools and software\Anaconda\envs\bookProjects\lib\site-packages\tensorflow_federated\python\learning\federated_averaging.py", line 211, in build_federated_averaging_process
stateful_delta_aggregate_fn, stateful_model_broadcast_fn)
File "B:\tools and software\Anaconda\envs\bookProjects\lib\site-packages\tensorflow_federated\python\learning\framework\optimizer_utils.py", line 498, in build_model_delta_optimizer_process
py_typecheck.check_callable(model_fn)
File "B:\tools and software\Anaconda\envs\bookProjects\lib\site-packages\tensorflow_federated\python\common_libs\py_typecheck.py", line 106, in check_callable
type_string(type(target))))
TypeError: Expected a callable, found non-callable tensorflow_federated.python.learning.model_utils.EnhancedModel.
highlighting:
in build_federated_averaging_process
stateful_delta_aggregate_fn, stateful_model_broadcast_fn)
and
TypeError: Expected a callable, found non-callable tensorflow_federated.python.learning.model_utils.EnhancedModel.
2 have tried
looked at another similar issue here have tried to make
model_fn a collection.abc Callable, model_fn=Callable[[], model_fn]
only creates a new error.
3 some code:
iterative process:
model_fn = model_builder(input_dim=sysarg,
input_spec=input_spec)
iterative_process = tff.learning.build_federated_averaging_process(
model_fn=model_fn,
client_optimizer_fn=client_optimizer_fn,
server_optimizer_fn=server_optimizer_fn,
)
iterative_process = compression_process_adapter.CompressionProcessAdapter(iterative_process)```
model bulder:
def model_builder(input_dim, input_spec):
model = create_model(input_dim)
return tff.learning.from_keras_model(keras_model=model,
loss=tf.keras.losses.MeanSquaredError(),
input_spec=input_spec,
metrics=[tf.keras.metrics.Accuracy()],
)
create model (for good measure)
def create_model(input_dim):
autoencoder = Sequential([
tf.keras.layers.Dense(int(0.75 * input_dim), activation="tanh", input_shape=(input_dim,)),
tf.keras.layers.Dense(int(0.5 * input_dim), activation="tanh"),
tf.keras.layers.Dense(int(0.33 * input_dim), activation="tanh"),
tf.keras.layers.Dense(int(0.25 * input_dim), activation="tanh"),
tf.keras.layers.Dense(int(0.33 * input_dim), activation="tanh"),
tf.keras.layers.Dense(int(0.5 * input_dim), activation="tanh"),
tf.keras.layers.Dense(int(0.75 * input_dim), activation="tanh"),
tf.keras.layers.Dense(input_dim)
])
The model_fn argument of tff.learning.build_federated_averaging_process needs to be a callable (What is a callable?) that takes no arguments and returns a tff.learning.Model.
From the code (reproduced here for readability):
def model_builder(input_dim, input_spec):
model = create_model(input_dim)
return tff.learning.from_keras_model(
keras_model=model,
loss=tf.keras.losses.MeanSquaredError(),
input_spec=input_spec,
metrics=[tf.keras.metrics.Accuracy()])
model_fn = model_builder(input_dim=sysarg, input_spec=input_spec)
iterative_process = tff.learning.build_federated_averaging_process(
model_fn=model_fn,
client_optimizer_fn=client_optimizer_fn,
server_optimizer_fn=server_optimizer_fn)
model_fn is actually an instance of tff.learning.Model, rather than a callable that constructs and returns a model.
model_builder should be passed to tff.learning.build_federated_averaging_process, but since it takes arguments this won't work as-is.
One alternative would be to use functools.partial:
iterative_process = tff.learning.build_federated_averaging_process(
model_fn=functools.partial(model_builder, input_dim=sysarg, input_spec=input_spec),
client_optimizer_fn=client_optimizer_fn,
server_optimizer_fn=server_optimizer_fn)
Or even use a no-arg lambda:
iterative_process = tff.learning.build_federated_averaging_process(
model_fn=lambda: model_builder(input_dim=sysarg, input_spec=input_spec),
client_optimizer_fn=client_optimizer_fn,
server_optimizer_fn=server_optimizer_fn)
The following question has a good discussion on the differences between lambdas and partials: Python: Why is functools.partial necessary?

Python SUDS not giving description of WebFault

I am creating all the attributes properly and I know that my request is complete because it is giving a TypeError when I add any extraneous fields that my web service does not accept. Now when I call the Method like client.service.Method(attributes...) I get a WebFault but there is no description for why the WebFault is occurring. Has anyone faced this problem before or does anyone know what this could mean?
The traceback is:
File "C:/Python25/sudsfirst.py", line 34, in <module>
result = client.service.Method(arg1,arg2)
File "build\bdist.win32\egg\suds\client.py", line 542 in __call__
return client.invoke(args, kwargs)
File "build\bdist.win32\egg\suds\client.py", line 602, in invoke
result = self.send(soapenv)
File "build\bdist.win32\egg\suds\client.py", line 649, in send
result = self.failed(binding, e)
File "build\bdist.win32\egg\suds\bindings\binding.py", line 265, in get_fault
raise WebFault(p, faultroot)
WebFault
The code is:
url = 'http://blah?wsdl'
client = Client(url)
arg1 = client.factory.create('type1')
arg1.type1 = ...
arg2 = client.factory.create('type2')
arg2.type1 = ...
client.service.Method(arg1, arg2)
Sorry, I can't mention more than this about the code.

Resources