I'm facing a little bit annoying problem. The tensorflow-federated training (initialize and next) takes a long time to start (I'm not talking about time to finish, it's just starting time takes a while).
I doubt that this is due to either using: 1) with eager_mode():,
or 2) use shuffling, as below:
with eager_mode():
def preprocess(new_dataset):
def map_fn(elem):
return collections.OrderedDict([('x', tf.reshape(elem['In'], [-1])),('y', tf.reshape(elem['Out'],[1]))])
DS2= new_dataset.map(map_fn)
if Use_shuffle:
return DS2.repeat(SNN_epoch).shuffle(shuffle_buffer).batch(SNN_batch_size)
else:
return DS2.repeat(SNN_epoch).batch(SNN_batch_size)
...
...
...
This is what I do:
trainer_Itr_Process = tff.learning.build_federated_averaging_process(model_fn_Federated,server_optimizer_fn=(lambda : tf.keras.optimizers.SGD(learning_rate=learn_rate)),client_weight_fn=None)
FLstate = trainer_Itr_Process.initialize()
# Track loss of different ...... of federated iteration
for round_num in range(Fed_iter_min,Fed_iter_max):
FLstate, FLoutputs = trainer_Itr_Process.next(FLstate, federated_train_data)
......
......
......
This is the warning I'm getting:
W0616 11:30:00.217065 139843447875392 deprecation_wrapper.py:118] From /usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/api/_v1/estimator/__init__.py:10: The name tf.estimator.inputs is deprecated. Please use tf.compat.v1.estimator.inputs instead.
W0616 11:30:02.400945 139843447875392 deprecation_wrapper.py:118] From /usr/local/lib/python3.6/dist-packages/tensorflow_federated/python/core/impl/tensorflow_serialization.py:296: The name tf.initializers.variables is deprecated. Please use tf.compat.v1.initializers.variables instead.
Related
In Machine Translation Dataset I have successfully pre-trained and trained my model in Lua. Now I move to predict my model.
But I get the error in a Lua file in the function encoder_clones:forward(encoder_input)
Error: attempt to call global 'forward' (a nil value)
This is that specific function :
function fwd_prop_enc(source, source_l, batch_l, train)
local rnn_state_enc = reset_state(init_fwd_enc, batch_l, 0)
--print(rnn_state_enc)
local context = context_proto[{{1, batch_l}, {1, source_l}}]
for t = 1, source_l do
if train then
encoder_clones[t]:training()
else
encoder_clones[t]:evaluate()
end
local encoder_input = {source[t], unpack(rnn_state_enc[t-1])}
local out = encoder_clones:forward(encoder_input)
print("yes")
rnn_state_enc[t] = out
context[{{},t}]:copy(out[#out]) -- copy final layer for t'th timestep (for entire batch)
end
return rnn_state_enc, context
end
This is how it is called in the main function
local rnn_state_enc, context = fwd_prop_enc(source, source_l, 1, false) -- train=false
That error means that the function encoder_clones.forward doesn't actually exist.
I do not know what framework you are using, but a quick search revealed that encoder_clones is probably an array. Looking at your code, I noticed that your reference to encoder_clones is inside a for loop for t = 1, source_l do. What happens if you change encoder_clones:forward to encoder_clones[t]:forward? This will probably solve your problem.
If this does not solve your problem, can you tell us what framework you are using? Is it OpenNMT? We can't help you much further unless we know what encoder_clones is and where it came from.
I have a NodeMCU ESP8266 board running MicroPython. I'm running a web server on my ESP8266. This is my first IoT project based on one of these boards.
The below is a snippet of the code.
This is being executed within main.py. Every now and then, something causes the code to crash (perhaps timing and request based). When main.py exits, for whatever reason, I'm dropped back at the python CLI.
I'd like for the board to reset when this happens (if there isn't a better way).
What is the best method of restarting/reseting the ESP8266?
addr = socket.getaddrinfo('0.0.0.0', 80)[0][-1]
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(addr)
s.listen(5)
print('listening on', addr)
while True:
cl, addr = s.accept()
print('client connected from', addr)
cl_file = cl.makefile('rwb', 0)
print("Request:")
while True:
line = cl_file.readline()
print("Line:" , line)
if not line or line == b'\r\n':
print("breaking")
break
if line == b'GET /active HTTP/1.1\r\n':
MicroPython has machine.reset() function to reset a board.
Python (not just MicroPython) uses exception handling to handle errors.
Combining the two, you can easily achieve what you want. For example:
a = 4
b = 2
try:
a / b
except:
machine.reset()
If in the code above you replace value of b with 0, your board will reset. If you think about it for a bit, you probably will find out that it doesn't make much sense - you don't want your board suddenly reset if you just divide by 0 by mistake or otherwise. There're got to be better ways to handle errors! Likewise, you may want to think about your own case and see if resetting the board is really the best choice. If you think that yes, that's fine, just always keep in mind that you programmed your board to suddenly reset. Otherwise, your next question here may be "My board suddenly resets! Why???" ;-)
It may be late for the original question, but the answer I am going to share might help other people. Consider it is not a final solution, but in many scenarios, it may save a day. You can explore your case.
The solution is using the internal scheduling function of MicroPython. since its execution is guaranteed, then its behavior can be used as a tool to mimic a functional watchdog.
Following code will run with given timers and threshold which can be customized in your case, and if the timer reaches its threshold, and the value of wd_buffer is not updated for that time, then the function might be called, and we repeat the process again.
So in order to prevent the ESP getting restarted in this case after 12 sec, you have to in someplace in your code, periodically (shorter than 12 sec or adjust the timer and threshold according to your need) update the value of the Global wd_buffer variable. Hope it helps.
# Simple WD - Global Variable
wd_feeder = 0
wd_buffer = 0
wd_counter = 0
wd_threshold = 4
def wd_checker(calledvalue):
print('watchdog is checking... feeder= {} buffer= {}'.format(wd_feeder, wd_buffer))
global wd_counter
global wd_buffer
global wd_feeder
if wd_feeder == wd_buffer:
print('state is suspicious ... counter is {} incrementing the counter'.format(wd_counter))
wd_counter += 1
else:
wd_counter = 0
wd_feeder = wd_buffer
if wd_counter == wd_threshold:
print('Counter is reached its threshold, following function will be called')
wd_feeder = wd_buffer = wd_counter = 0
machine.reset()
if __name__ == '__main__':
scheduler_wd = machine.Timer(-1)
scheduler_wd.init(period=3000, mode=machine.Timer.PERIODIC, callback=wd_checker)
you could add a while loop checking for the Flash Button (GPIO pin 0) like this:
import machine
pin = machine.Pin(0, machine.Pin.IN, machine.Pin.PULL_UP)
while pin.value():
print('Put your code here...')
print('..this will looping until the Flash button is pressed...')
print('...and then it continues here.')
You could execute your code (which should be outside of the main.py -> other file) from the boot or the main.py. if it drops out it should execute the following code, which could trigger a reset.
You may have to catch the error first.
I hope I helped
This code is for a modding engine, Unitale base on Unity Written in Lua
So I am trying to use a Boolean Variable in my script poseur.lua, so when certain conditions are met so I can pass it to the other script encounter.lua, where a engine Predefined functions is being uses to make actions happens base on the occurring moment.
I tried to read the engine documentation multiple times, follow the exact syntax of Lua's fonction like GetVar(), SetVar(), SetGobal(),GetGlobal().
Searching and google thing about the Language, post on the subreddit and Game Exchange and tried to solve it by myself for hours... I just can't do it and I can't understand why ?
I will show parts of my codes for each.
poseur:
-- A basic monster script skeleton you can copy and modify for your own creations.
comments = {"Smells like the work\rof an enemy stand.",
"Nidhogg_Warrior is posing like his\rlife depends on it.",
"Nidhogg_Warrior's limbs shouldn't\rbe moving in this way."}
commands = {"GREET", "JUMP", "FLIRT", "CRINGE"}
EndDialougue = {" ! ! !","ouiii"}
sprite = "poseur" --Always PNG. Extension is added automatically.
name = "Nidhogg_Warrior"
hp = 99
atk = 1
def = 1
check = "The Nidhogg_Warrior is\rsearching for the Nidhogg"
dialogbubble = "rightlarge" -- See documentation for what bubbles you have available.
canspare = false
cancheck = true
GreetCounter = 5
Berserk = false
encounter:
-- A basic encounter script skeleton you can copy and modify for your own creations.
encountertext = "Nidhogg_Warrior is\rrunning frantically"
nextwaves = {"bullettest_chaserorb"}
wavetimer = 5.0
arenasize = {155, 130}
music = "musAncientGuardian"
enemies = {"poseur"}
require("Monsters.poseur")
enemypositions = {{0, 0}}
-- A custom list with attacks to choose from.
-- Actual selection happens in EnemyDialogueEnding().
-- Put here in case you want to use it.
possible_attacks = {"bullettest_bouncy", "bullettest_chaserorb", "bullettest_touhou"}
function EncounterStarting()
-- If you want to change the game state immediately, this is the place.
Player.lv = 20
Player.hp = 99
Player.name = "Teemies"
poseur.GetVar("Berserk")
end
Thank you for reading.
The answer to my problem was to use SetGobal(), GetGobal().
For some reasons my previous attempt to simply use SetGobal()Resulted in nil value despite writing it like that SetGobal("Berserk",true) gave me a nill value error, as soon as I launch the game.
But I still used them wrong. First I needed to put it SetGobal() at the end of the condition instead of at the start of the the poseur.lua script because the change of value... for some reasons was being overwritten by it first installment.
And to test the variable in the function in my encounter.lua, I needed to write it like that
function EnemyDialogueStarting()
-- Good location for setting monster dialogue depending on how the battle is going.
if GetGlobal("Jimmies") == true then
TEEEST()
end
end
Also any tips an suggestions are still welcome !
Well firstly, in lua simple values like bool and number are copied on assignment:
global={}
a=2
global.a=a--this is a copy
a=4--this change won't affect value in table
print(global.a)--2
print(a)--4
Secondly,
SetGobal and the other mentioned functions are not part of lua language, they must be related to your engine. Probably, they use word 'Global' not as lua 'global' but in a sense defined by engine.
Depending on the engine specifics these functions might as well do a deep copy of any variable they're given (or might as well not work with complicated objects).
I'm trying to get zipline working with non-US, intraday data, that I've loaded into a pandas DataFrame:
BARC HSBA LLOY STAN
Date
2014-07-01 08:30:00 321.250 894.55 112.105 1777.25
2014-07-01 08:32:00 321.150 894.70 112.095 1777.00
2014-07-01 08:34:00 321.075 894.80 112.140 1776.50
2014-07-01 08:36:00 321.725 894.80 112.255 1777.00
2014-07-01 08:38:00 321.675 894.70 112.290 1777.00
I've followed moving-averages tutorial here, replacing "AAPL" with my own symbol code, and the historical calls with "1m" data instead of "1d".
Then I do the final call using algo_obj.run(DataFrameSource(mydf)), where mydf is the dataframe above.
However there are all sorts of problems arising related to TradingEnvironment. According to the source code:
# This module maintains a global variable, environment, which is
# subsequently referenced directly by zipline financial
# components. To set the environment, you can set the property on
# the module directly:
# from zipline.finance import trading
# trading.environment = TradingEnvironment()
#
# or if you want to switch the environment for a limited context
# you can use a TradingEnvironment in a with clause:
# lse = TradingEnvironment(bm_index="^FTSE", exchange_tz="Europe/London")
# with lse:
# the code here will have lse as the global trading.environment
# algo.run(start, end)
However, using the context doesn't seem to fully work. I still get errors, for example stating that my timestamps are before the market open (and indeed, looking at trading.environment.open_and_close the times are for the US market.
My question is, has anybody managed to use zipline with non-US, intra-day data? Could you point me to a resource and ideally example code on how to do this?
n.b. I've seen there are some tests on github that seem related to the trading calendars (tradincalendar_lse.py, tradingcalendar_tse.py , etc) - but this appears to only handle data at the daily level. I would need to fix:
open/close times
reference data for the benchmark
and probably more ...
I've got this working after fiddling around with the tutorial notebook. Code sample below. It's using the DF mid, as described in the original question. A few points bear mentioning:
Trading Calendar I create one manually and assign to trading.environment, by using non_working_days in tradingcalendar_lse.py. Alternatively you could create one that fits your data exactly (however could be a problem for out-of-sample data). There are two fields that you need to define: trading_days and open_and_closes.
sim_params There is a problem with the default start/end values because they aren't timezone aware. So you must create a sim_params object and pass start/end parameters with a timezone.
Also, run() must be called with the argument overwrite_sim_params=False as calculate_first_open/close raise timestamp errors.
I should mention that it's also possible to pass pandas Panel data, with fields open,high,low,close,price and volume in the minor_axis. But in this case, the former fields are mandatory - otherwise errors are raised.
Note that this code only produces a daily summary of the performance. I'm sure there must be a way to get the result at a minute resolution (I thought this was set by emission_rate, but apparently it's not). If anybody knows please comment and I'll update the code.
Also, not sure what the api call is to call 'analyze' (i.e. when using %%zipline magic in IPython, as in the tutorial, the analyze() method gets automatically called. How do I do this manually?)
import pytz
from datetime import datetime
from zipline.algorithm import TradingAlgorithm
from zipline.utils import tradingcalendar
from zipline.utils import tradingcalendar_lse
from zipline.finance.trading import TradingEnvironment
from zipline.api import order_target, record, symbol, history, add_history
from zipline.finance import trading
def initialize(context):
# Register 2 histories that track daily prices,
# one with a 100 window and one with a 300 day window
add_history(10, '1m', 'price')
add_history(30, '1m', 'price')
context.i = 0
def handle_data(context, data):
# Skip first 30 mins to get full windows
context.i += 1
if context.i < 30:
return
# Compute averages
# history() has to be called with the same params
# from above and returns a pandas dataframe.
short_mavg = history(10, '1m', 'price').mean()
long_mavg = history(30, '1m', 'price').mean()
sym = symbol('BARC')
# Trading logic
if short_mavg[sym] > long_mavg[sym]:
# order_target orders as many shares as needed to
# achieve the desired number of shares.
order_target(sym, 100)
elif short_mavg[sym] < long_mavg[sym]:
order_target(sym, 0)
# Save values for later inspection
record(BARC=data[sym].price,
short_mavg=short_mavg[sym],
long_mavg=long_mavg[sym])
def analyze(context,perf) :
perf["pnl"].plot(title="Strategy P&L")
# Create algorithm object passing in initialize and
# handle_data functions
# This is needed to handle the correct calendar. Assume that market data has the right index for tradeable days.
# Passing in env_trading_calendar=tradingcalendar_lse doesn't appear to work, as it doesn't implement open_and_closes
from zipline.utils import tradingcalendar_lse
trading.environment = TradingEnvironment(bm_symbol='^FTSE', exchange_tz='Europe/London')
#trading.environment.trading_days = mid.index.normalize().unique()
trading.environment.trading_days = pd.date_range(start=mid.index.normalize()[0],
end=mid.index.normalize()[-1],
freq=pd.tseries.offsets.CDay(holidays=tradingcalendar_lse.non_trading_days))
trading.environment.open_and_closes = pd.DataFrame(index=trading.environment.trading_days,columns=["market_open","market_close"])
trading.environment.open_and_closes.market_open = (trading.environment.open_and_closes.index + pd.to_timedelta(60*7,unit="T")).to_pydatetime()
trading.environment.open_and_closes.market_close = (trading.environment.open_and_closes.index + pd.to_timedelta(60*15+30,unit="T")).to_pydatetime()
from zipline.utils.factory import create_simulation_parameters
sim_params = create_simulation_parameters(
start = pd.to_datetime("2014-07-01 08:30:00").tz_localize("Europe/London").tz_convert("UTC"), #Bug in code doesn't set tz if these are not specified (finance/trading.py:SimulationParameters.calculate_first_open[close])
end = pd.to_datetime("2014-07-24 16:30:00").tz_localize("Europe/London").tz_convert("UTC"),
data_frequency = "minute",
emission_rate = "minute",
sids = ["BARC"])
algo_obj = TradingAlgorithm(initialize=initialize,
handle_data=handle_data,
sim_params=sim_params)
# Run algorithm
perf_manual = algo_obj.run(mid,overwrite_sim_params=False) # overwrite == True calls calculate_first_open[close] (see above)
#Luciano
You can add analyze(None, perf_manual)at the end of your code for automatically running the analyze process.
We are running application in Spring context using DataNucleus as our ORM mapping and mysql as our database.
Our application have a daily import job of some data feed into our database. The size of the data feed translate into around 1 millions row of insert/update. The performance of the import start out to be very good but then it degrade overtime (as the number of executed query increase) and at some point the application freeze or stop responding. We will have to wait for the whole job to complete before the application response again.
This behavior looks very like a memory leak to us and we have been looking hard at our code to catch any potential problem, however the problem didn't go away. One interesting thing we found from the heap dump is that org.datanucleus.ExecutionContextThreadedImpl (or the HashSet/HashMap) hold 90% of our memory (5GB) during the import. (I have attahed the screenshot of the dump below). My research on the internet said this reference is the Level1 Cache (not sure am i correct). My question is during a large import, how can i limit/control the size of the level1 cache. May be ask DN to not cache during my import?
If that's not the L1 cache, what's the possible cause of my memory issue?
Our code use a transaction for every insert to prevent locking of large chunk of data in the database. It's call the flush method every 2000 insert
As a temporary fix, we moved our import process to run overnight when no one is using our app. Obviously, this cannot go on forever. Please could someone at least point us in the right direction so that we can do more research and hoping we can find a fixes.
Would be good if someone have knowledge of decoding the heap dump
Your help would be very much appreciated by all of us here. Many thanks!
https://s3-ap-southeast-1.amazonaws.com/public-external/datanucleus_heap_dump.png
https://s3-ap-southeast-1.amazonaws.com/public-external/datanucleus_dump2.png
Code Below - Caller of this method does not have a transaction. This method will process one import object per call, and we need to process around 100K of these object daily
#Override
#PreAuthorize("(hasUserRole('ROLE_ADMIN')")
#Transactional(propagation = Propagation.REQUIRED)
public void processImport(ImportInvestorAccountUpdate account, String advisorCompanyKey) {
ImportInvestorAccountDescriptor invAccDesc = account
.getInvestorAccount();
InvestorAccount invAcc = getInvestorAccountByImportDescriptor(
invAccDesc, advisorCompanyKey);
try {
ParseReportingData parseReportingData = ctx
.getBean(ParseReportingData.class);
String baseCCY = invAcc.getBaseCurrency();
Date valueDate = account.getValueDate();
ArrayList<InvestorAccountInformationILAS> infoList = parseReportingData
.getInvestorAccountInformationILAS(null, invAcc, valueDate,
baseCCY);
InvestorAccountInformationILAS info = infoList.get(0);
PositionSnapshot snapshot = new PositionSnapshot();
ArrayList<Position> posList = new ArrayList<Position>();
Double totalValueInBase = 0.0;
double totalQty = 0.0;
for (ImportPosition importPos : account.getPositions()) {
Asset asset = getAssetByImportDescriptor(importPos
.getTicker());
PositionInsurance pos = new PositionInsurance();
pos.setAsset(asset);
pos.setQuantity(importPos.getUnits());
pos.setQuantityType(Position.QUANTITY_TYPE_UNITS);
posList.add(pos);
}
snapshot.setPositions(posList);
info.setHoldings(snapshot);
log.info("persisting a new investorAccountInformation(source:"
+ invAcc.getReportSource() + ") on " + valueDate
+ " of InvestorAccount(key:" + invAcc.getKey() + ")");
persistenceService.updateManagementEntity(invAcc);
} catch (Exception e) {
throw new DataImportException(invAcc == null ? null : invAcc.getKey(), advisorCompanyKey,
e.getMessage());
}
}
Do you use the same pm for the entire job?
If so, you may want to try to close and create new ones once in a while.
If not, this could be the L2 cache. What setting do you have for datanucleus.cache.level2.type? It think it's a weak map by default. You may want to try none for testing.