I want to have a single script, that either collects tensorboard data or not, depending on how I run it. I am aware that I can pass flags to tell my script how I want it to be run. I could even hard code it in the script and just manually change the script.
Either solution has a bigger problem. I find myself having to write an if statement everywhere on my script when I want the summary writer operations to be ran or not. For example I find that I would have to do something like:
if tb_sys_arg = 'tensorboard':
merged = tf.merge_all_summaries()
and then depending on the value of tb_sys_arg run the summaries or not, as in:
if tb_sys_arg = 'tensorboard':
merged = tf.merge_all_summaries()
else:
train_writer = tf.train.SummaryWriter(tensorboard_data_dump_train, sess.graph)
this seems really silly to me. I'd rather not have to do that. Is this the right way to do this? I just don't want to collect statistics each time I run my main script but I also don't want to have two separate scripts either.
As an anecdotical story, few months ago I started using TensorBoard and it seems I have been running my main file as follow:
python main.py —logdir=/tmp/mdl_logs
so that it collects tensorboard data. But realized that I don't think I need that last flag to collect tensorboard data. Its been so long that now I forget if I actually need that. I've been reading the documentation and tutorials but it seems I don't need that last flag (its only needed to run the web app as in tensorboard --logdir=path/to/log-directory, right?) Have I been doing this wrong all this time?
You can launch Supervisor without "summary" service, so it won't run the summary nodes, see "Launching fewer services" section of the Supervisor docs -- https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/functions_and_classes/shard6/tf.train.Supervisor.md#launching-fewer-services
Related
Context
So I finally give a try to Fish, and as one would expect I encounter some frictions due to differences with my usual routines.
The most astonishing for me, as for many other, was the absence of the bang operator. I'm fine with the lose of sudo !!, as the suggested function replacement seems even better to me, I named it gar which means "To make, compel (someone to do something); to cause (something to be done." However I'll need a replacement for !<abc><enter> which grab the last history line starting with <abc> and run it without further ado, suggestions are welcome.
Now, for the more personal things:
- I use a Typematrix 2030 keyboard
- I use a bépo layout
- I like to configure default finger position keys with the most used actions
Aims
As on my keybord <enter> is well positioned and is semantically relevant for that, ideally I would like to achieve the following key binding:
ctrl-enter: accept the whole suggestion and run it without further confirmation
ctrl-tab: accept the whole suggestion and wait for further edit
alt-enter: redo the last command without further confirmation
But according to xev it appears that, at least with Gnome-terminal, this combinations are not recognized. Are they terminal that supports it? For now I remapped these three to <ctrl>-i, <alt>-i and <alt>-I respectively:
bind --preset \ci forward-char execute
bind --preset \ei forward-char
bind --preset \eI forward-word
This works as expected, but it seems that now the tab key will also map to the first item. I guess that tab map to <alt>-i at some point in the shell stack. I wasn't aware of that, so I don't know yet if it will be possible for Fish to separate each of them.
To manage jobs, I also came with
bind --preset \es fg
bind --preset \eS bg
The first works as expected, but the second one doesn't. With application like vim, the binding should be operated in the application configuration itself of course. But for things as trivial as yes, <alt>-S won't work as expected while <crl>-z continue to operate normally.
I also would like to bind some commands like ls -alh and git status --short to a directly executed command, showing the result bellow the currently edited line, allowing to further type seamlessly, but didn't find the way to do it yet.
Summary of remaining question
So here are my more precise questions summarised:
how do I bind the sleep signal to <alt>-S?
is there a terminal I can use where <alt>-<enter> and <ctrl>-<enter> works?
how to seamlessly run command while maintaining the current line edition in place?
can you bind something to <alt>-i without altering <tab>?
how do I bind the sleep signal to -S?
What you are doing with bind \es fg is to alter a binding inside the shell.
But when you execute yes, the shell isn't currently in the foreground, so shell bindings don't apply.
What you'd have to do instead is change the terminal settings via stty susp \cs,
but fish resets the terminal settings when executing commands (so you can't accidentally break them and end up in an unusable environment), so there currently is no way to do this in fish.
can you bind something to <alt>-i without altering <tab>?
Sure. You bind \ei. Which is escape+i, which is alt-i (because in a terminal alt is escape).
Your problem is with ctrl-i, which in the way terminals encode control+character is tab. The application receives an actual tab character, and at that point the information has been lost.
is there a terminal I can use where - and - works?
Most terminals should send \e\r for alt-enter. ctrl-enter again is unencodable with the usual code (because \r is ctrl-m), just like ctrl-tab is.
Any fix to this requires the terminal to encode these combination differently.
how to seamlessly run command while maintaining the current line edition in place?
I don't know what you mean by this. I'm guessing you want fish to remain open and editable while a command also runs in the foreground. That can't work. There's no way to synchronize output from two commands to a terminal, not with cursor movement being what it is.
Let's say I have a rule like this.
foo(
name = "helloworld",
myarray = [
":bar",
"//path/to:qux",
],
)
In this case, myarray is static.
However, I want it to be given by cli, like
bazel run //:helloworld --myarray=":bar,//path/to:qux,:baz,:another"
How is this possible?
Thanks
To get exactly what you're asking for, Bazel would need to support LABEL_LIST in Starlark-defined command line flags, which are documented here:
https://docs.bazel.build/versions/2.1.0/skylark/lib/config.html
and here: https://docs.bazel.build/versions/2.1.0/skylark/config.html
Unfortunately that's not implemented at the moment.
If you don't actually need a list of labels (i.e., to create dependencies between targets), then maybe STRING_LIST will work for you.
If you do need a list of labels, and the different possible values are known, then you can use --define, config_setting(), and select():
https://docs.bazel.build/versions/2.1.0/configurable-attributes.html
The question is, what are you really after. Passing variable, array into the bazel build/run isn't really possible, well not as such and not (mostly) without (very likely unwanted) side effects. Aren't you perhaps really just looking into passing arguments directly to what is being run by the run? I.e. pass it to the executable itself, not bazel?
There are few ways you could sneak stuff in (you'd also in most cases need to come up with a syntax to pass data on CLI and unpack the array in a rule), but many come with relatively substantial price.
You can define your array in a bzl file and load it from where the rule uses it. You can then dump the bzl content rewriting your build/run configuration (also making it obvious, traceable) and load the bits from the rule (only affecting the rule loading and using the variable). E.g, BUILD file:
load(":myarray.bzl", "myarray")
foo(
name = "helloworld",
myarray = myarray,
],
)
And you can then call your build:
$ echo 'myarray=[":bar", "//path/to:qux", ":baz", ":another"]' > myarray.bzl
$ bazel run //:helloworld
Which you can of course put in a single wrapper script. If this really needs to be a bazel array, this one is probably the cleanest way to do that.
--workspace_status_command: you can collection information about your environment, add either or both of the resulting files (depending on whether the inputs are meant to invalidate the rule results or not, you could use volatile or stable status files) as a dependency of your rule and process the incoming file in the what is being executed by the rule (at which point one would wonder why not pass it to as its command line arguments directly). If using stable status file, also each other rule depending on it is invalidated by any change.
You can do similar thing by using --action_env. From within the executable/tool/script underpinning the rule, you can directly access defined environmental variable. However, this also means environment of each rule is affected (not just the one you're targeting); and again, why would it parse the information from environment and not accept arguments on the command line.
There is also --define, but you would not really get direct access it's value as much as you could select() a choice out of possible options.
I'm building some simple load testing for my API, and to make sure everything is on the up and up I'd like to also review the response headers and data. But when I run my test using the command line and then re-open the GUI to add a View Results Tree listener and load the created file the response headers or response data is empty.
I entered the following values into user.properties (also tried uncommenting those values in jmeter.properties and changing them there, same result)
jmeter.save.saveservice.output_format=csv (tried xml, omitting it, jtl)
jmeter.save.saveservice.data_type=false
jmeter.save.saveservice.label=true
jmeter.save.saveservice.response_code=true
jmeter.save.saveservice.response_data.on_error=true
jmeter.save.saveservice.response_message=true
jmeter.save.saveservice.successful=true
jmeter.save.saveservice.thread_name=true
jmeter.save.saveservice.time=true
jmeter.save.saveservice.subresults=false
jmeter.save.saveservice.assertions=false
jmeter.save.saveservice.latency=true
jmeter.save.saveservice.bytes=true
jmeter.save.saveservice.hostname=true
jmeter.save.saveservice.thread_counts=true
jmeter.save.saveservice.sample_count=true
jmeter.save.saveservice.response_message=true
jmeter.save.saveservice.assertion_results_failure_message=true
jmeter.save.saveservice.timestamp_format=HH:mm:ss
jmeter.save.saveservice.default_delimiter=;
jmeter.save.saveservice.print_field_names=true
But still no luck when opening the result file. I tried declaring the file after the -l tag as results.csv, .jtl, even .xml but none of them show me the headers and data.
I'm running it locally on Mac OS X 10.10 using the following command, jmeter version is 2.12
java -jar ApacheJMeter.jar -n -t /Users/[username]/Documents/API_test.jmx -l results_15.jtl
I don't know if it's not even saving that data, or if the Listeners can't read it or if I've been cursed but any help is appreciated.
It works fine if I add a Listener and run it using the GUI, but if I try to run my larger tests that way, well, things don't end well for anyone.
So my question is:
How do I save the response header and data to a file when using the command line, and how do I then view said file in jmeter?
Add a Simple Data Writer (under Listeners) and output to a file (NB: different file than your log). Under the 'configure' button, there are all sorts of options of what to save. One of the check boxes is Save Response Header.
This file can get huge if you're saving a bunch of things for every request- one strategy is to check everything, but only save for errors. But you can do whatever works for you.
You can also turn on "Functional Test Mode" which will produce a large file but will contain pretty much anything you might need to debug your test.
Beware, this can create a very large JTL file, so don't forget to turn it off for your large test runs! See JMeter Maven mojo throws IllegalArgumentException with large JTL file
Alternatively use a Tree View Listener in the GUI for a small sample of the requests and check the request/response in the GUI (including headers) to debug or check your test.
Add Below lines in user.properties file
jmeter.save.saveservice.output_format=xml
jmeter.save.saveservice.response_data=true
jmeter.save.saveservice.samplerData=true
jmeter.save.saveservice.requestHeaders=true
jmeter.save.saveservice.url=true
Restart cmd prompt.
I have been scouring the internet for days, I have a problem similar to this.
I need to retrieve the console output in raw (plain) text. But if I can get it in HTML that is fine too, I can always parse it. The only thing is that I need to get it during the build step, which is a problem since the location where it should be available is truncated...
I have tried retrieving the console output from the following URL's (relative to the job):
/consoleText
/logText/progressiveText
/logText/progressiveHTML
The two text ones are plain text and would be perfect if not for the truncation, same goes for the HTML one... exactly what I need - only its truncated....
I am sure it is possible to retrieve this information somehow, since when viewing /consoleFull there is a real-time update of the console, without truncating or buffering.
However, upon examining that web page, instead of finding the content I desired, I found this code where it should have been (I did not include the full pages code, since it would be mostly irrelevant, and I believe those answering would be able to find out and know what should be there on their own)
new Ajax.Request(href,{
method: "post",
parameters: {"start":e.fetchedBytes},
requestHeaders: headers,
onComplete: function(rsp,_) {
var stickToBottom = scroller.isSticking();
var text = rsp.responseText;
if(text!="") {
var p = document.createElement("DIV");
e.appendChild(p); // Needs to be first for IE
// Use "outerHTML" for IE; workaround for:
// http://www.quirksmode.org/bugreports/archives/2004/11/innerhtml_and_t.html
if (p.outerHTML) {
p.outerHTML = '<pre>'+text+'</pre>';
p = e.lastChild;
}
else p.innerHTML = text;
Behaviour.applySubtree(p);
if(stickToBottom) scroller.scrollToBottom();
}
e.fetchedBytes = rsp.getResponseHeader("X-Text-Size");
e.consoleAnnotator = rsp.getResponseHeader("X-ConsoleAnnotator");
if(rsp.getResponseHeader("X-More-Data")=="true")
setTimeout(function(){fetchNext(e,href);},1000);
else
$("spinner").style.display = "none";
}
});
Specifically, I am hoping there is a way for me to get the content from text whatever it may be. I am not familiar with this language and so am not sure how I might be able to get the content I want. Plugins won't help since I want to retrieve this content as part of my script during the build step
You did pretty much good investigation already. I can only add the following: all console related plug-ins I know are designed as a post build actions.
The Log Trigger plugin provides a post-build action that allows Hudson
builds to search their console log for a given regular expression and
if found, trigger additional downstream jobs.
So it looks like there is no straightforward solution to your problem. I can see the following options:
1. Use tee or something similar (applicable to shell build steps only)
This solution is far from being universal, but it can provide quick access to the latest console output, produced by a command or set of command.
tee - read from standard input and write to standard output and files
Using synonyms on the system level other Jenkins build steps can modified in order to produce console output. File with console output can be referenced through Jenkins or using any other way.
2. Modify Jenkins code
You can just do a quick fix for internal usage or provide a patch introducing specific system-wide setting.
3. Mimic /console behavior
Code in your example is used to request updates from the Jenkins server. As you may expect the server side can return piece of information starting with some offset. Let me show.
Periodically console page sends requests to the server:
Parameters are straightforward:
Response is a chunk of information to be added:
Another request with updated offset (start) value
You can easily understand there is no data by analyzing Content-Length
So the answer is: use url/job-name/build-number/logText/progressiveHtml, specify start offset, send request and receive console update.
I had a similar issue, the last part of my Jenkinsfile build script needs to parse the ConsoleLog for particular error messages to put in an email build report.
First attempt: http request.
It felt like a hack, it mostly worked, but ran into issues when we locked down access to the Jenkins server & my build nodes could no longer perform annon http gets on the page
Second attempt: use the APIs to enumerate the log lines.
It felt like the right thing to do, but it failed horribly as my nodes would take 30 minutes to get through the 100 meg log files. My presumption is that the Jenkins server was not caching the file, so each request involved a re-reading of the entire file up until the point of the last read.
Third and most successful solution: run grep on the server.
node('master') {
sh 'grep some_criteria $JENKINS_HOME/workspace/path/to/job/console.log'
}
it was fast, reliable, and it didn't matter how big the log files were.
Yes, this required trust of the Jenkins admin and knowledge of the directory paths on the Jenkins server - but since I was the admin, I trusted myself to do the right thing. Your mileage may vary.
To add some insight: when the Jenkins build was in progress, the response for the .../consoleText URL maxed out at 10000 lines, exactly.
I was using 'requests()' package in Python. I have tried the same URL with curl and again received only the first 10K lines.
Only after the build has finished both methods returned the full log (>22K lines in my case).
I will research further and hope to report back.
[2015-08-18] Update: It seems that this is a known issue (see here) and it's fixed in Jenkins 1.618 and later. I am still running 1.615 so I cannot verify.
Amir
I have a Jenkins server that builds/tests about 50 projects. Unfortunately, some of these builds fail, but I don't have a good way to measure whether build failures are increasing or decreasing in frequency over time.
What I'd like is something along these lines:
A report that shows me, over the course of a month, how many jobs were unstable/failed
A report that says "X Days without a broken build" (kind of like at construction sites)
A "Red/Green calendar", that would show on a per-day basis whether any builds were broken
I didn't see any plugins that visualized data in any of these ways, but I'm willing to scrape the Jenkins logs to get the information. Is there a better way to see data similar to this?
I think this work pretty decent using the API. You can get all jobs from your view, then go into the job details and get the build numbers and build date. With those build numbers you can get the corresponding status. You would have to do some coding to collect and display the data, but this would be a possible way.
Another possibility would be using a Groovy script over the console in Manage Jenkins. I do not have much experience working with that feature though, but as you have access to the internal representation it should be pretty easy to get some data out of there.
Finally, the optimal solution would be to write a plugin that does the work, but this is of course also the solution that requires the most effort and know-how.
The Global Build Stats plugin might provide the reporting you're looking for.
(And if you already considered this plugin, I'm curious what problems you ran into.)
As #pushy mentions, the Groovy script console is a good tool to use for these types of statistics gathering. You can use the groovy script in the remote API as well. Here is a starting point for gathering information from all jobs matching a pattern.
def jobPattern='pattern'
Hudson.instance.getItems(Project).each {project ->
def results = [:]
if (project.name.contains(jobPattern)) {
results."$project.name" = [SUCCESS:0,UNSTABLE:0,FAILURE:0,ABORTED:0]
def build = project.getLastBuild()
while (build){
//println "$project.name;$build.id;$build.result"
results."$project.name"."$build.result" = results."$project.name"."$build.result" +1
build=build.getPreviousBuild()
}
}
results.each{name,map->
map.each{result,count->
println "$name : $result = $count"
}
}
}
"Done"
Use this as a start and modify according to your specific requirements.
Try build metric plugin along with Global Build Stat plugin.