This question requires a bit of backstory... At my company, we produce a set of PDF and HTML files. A very large set. The current build process (which I designed, in haste) is a Perl script that reads a set of files, where each file contains a new ant command to execute.
It is designed terribly.
Now, I'm trying to shift the entire project over to using ant for the majority of the tasks. Within a target, I can construct a list of files that need to be built, as either PDF or HTML. However, when I call the ant command to build each file, after about three builds (of, say, five), the entire process crashes with an OutOfMemory error. Furthermore, my buildlog.xml ends up being something like 20 megs--it concatenates every ant command's output into one giant log, since they are being called from a single target. With the earlier Perl solution, I was able to get a buildlog.xml for each ant command--simply save and rename the buildlog to something else.
Even if I set ant or java heap sizes in my user.properties, I still fail with an OOM eventually. I wonder if an appropriate solution is to call <exec> to launch a script that does some of what I described and desire: namely, call ant, rename the buildlog, and die--theoretically allocating and freeing up space better than one "giant" ant call. I am worried that I am going to be heading down another "hacky" solution to a problem that's well-defined, and can be entirely confined to ant. Then again, <exec> does exist for a reason, so should I not feel bad for using it?
As with most corporate software (at least those which have deadlines and, if yours don't, please let me know where you work so I can try get a job there), the first step is to get it working.
Then, worry about getting it working well.
For that first step, you can use any tool at your disposal, no matter how ugly you think it looks.
But you might want to make sure that the powers-that-be know that you've had to do all sorts of kludgy things to get it working for them, so that they allow you to hopefully fix it up before maintenance has to start on it. You probably don't want to be maintaining a hideously ugly code base or design.
We've unleashed such wonders on the world as applications that shut themselves down nightly to avoid memory leaks (leaving the OS to restart them), putting "questionable" code at the other end of a TCP socket so their crashing doesn't bring down the main application and, I'm sure, many other horrors that my brain has decided to remove all trace of.
Related
I want to evaluate the performance of Rascal for a given rewrite system that I've written. I'm wondering if there's a good way of doing it?
Ideally, I'd generate some compiled Java classes from the system and then run them manually against my inputs. Is there an easy or recommended way to do it?
Cheers,
One way to do this is to use the functions in the library util::Benchmark. Typically, you could write something like
cpuType( (){ call_the_function_I_want_to_observe(); } ). This will execute your function and print the cpu time used.
Note that Rascal can be executed in two ways: interpreted and compiled which makes a big difference when measuring performance. We are working hard at the moment to fully integrate the compiler in the Eclipse IDE, but a stand alone version is available as well. This can be called as java -Xss8m -jar rascal-0.8.4-SNAPSHOT.jar --compiledREPL followed by at least values for directories for sources (--src), and binaries (--bin). Here rascal-0.8.4-SNAPSHOT.jar (but most likely named differently) is downloaded from the https://update.rascal-mpl.org/console/rascal-shell-unstable.jar.
If you need more information, don't hesitate to ask for more details: this part of our tool chain is unfortunately still undocumented.
I have a project in Delphi7. Its is rather large consisting of 40 odd forms and frames.
Recently, the compiler only allows me to compile the project once so i can run it, then every re-compile the IDE hangs and i have to end the Delphi process. Before this occurs, my CPU goes to 50% (on dual core machine) so my deduction is the compilation process has gone into an infinite loop. The Executable it produces is not runnable and usually at a fixed size after it hangs.
I was wondering how i can go about finding where this inconsistency in my project is. Other projects do not suffer from this same issue.
You can use Process Explorer to discover what compiler is doing (reading a file, or ...).
Check the QC 3807 issue.
Check the system resources - free disk space, memory. Clean the temp folder. Check the disk for errors. Do you have antivirus running ? If yes, then try to turn it off.
Use "process of elimination", to see if it's something in your code.
First, make a backup of where you are, or save to your CVS (you ARE using version control, right? RIGHT? good.) Revert your branch to an earlier version where it worked. See if that works. If so, merge half of the changes from the present-day version. If that works, try the other half. Keep cutting things in half, and you'll find the code that causes the problem, by process of elimination.
Or, it may turn out to be something in the configuration. Carbonite may be your friend here.
You can either:
Enable "Compilation progress display" in the "Environment Options" window, in the "Preferences" tab.
Use the command line compiler bcc32.exe to have a detailed console output.
Both will let you know which file is hanging the compiler.
Take a look at the great Delphi Speed Up tool, which allows e.g. to abort CodeCompletion and HelpInsight by ESC/mouse move.
Our team had been using Delphi 6 for many years, then switched to Delphi 2006 years ago. With both versions we have the following problem: frequently the compiler complains about a unit which is supposedly used recursively. This unit is a 40k LOC unit which is at the core of a project with almost 1 million LOC (third party included).
The error message is incorrect: a full build on the project always works. Unfortunately, the error message does not tell us where the supposed circular reference is, just the name of that unit. Sometimes it even happens that valid error messages are listed 2-4 times until that circular reference problem is "found". Clearly the compiler is running in a circle here. Because of the size of that project it is hard to find the problem manually. Therefore I made a tool which proves that there really is no circular reference (the tool creates a directed dependency graph of the units and determines the coherence components in that graph - there are none except if I deliberately put some in).
This is not only affecting F9 compilation but also code completion / insight which is not working most of the time. Sometimes it works when I press ctrl-space a second time...
Any ideas how we can isolate or even fix the problem? Note that it will be very hard to split the 40k LOC unit into smaller ones because it contains about 15 large classes which depend on each other in the interface section (I know it's bad but should work anyway).
Update
We are constantly refactoring but this is one tough unit to refactor because everything depends on everything, almost. Have been trying to get around it via interfaces but we are talking about some classes with 100s of methods and properties. And it would be slower.
Upgrading to D2009 may be an option down the road but right now we're stuck with D2006 (the unicode stuff and the price tag are two of the stoppers here). Question is anyway if it would help since the problem is in there since D6 at least.
About trimming the uses clauses, we have been doing this frequently with Icarus. But that did not help so far. We are down to 90 custom units in the interface section now. However, with a true circular reference the problem could be in any unit. Also tried to add all units to the dpr.
The project shares a lot of code with other projects, and there are some IFDEFs. However, the defines are not set up in project options but via a common include file. Therefore all modules should see the same defines. Also, the problem reoccurs shortly after a full rebuild without switching to another project.
I will probably be downvoted for this. In D2005 I had a 10k loc unit (datamodule) that flat out stopped compiling. Had to separate out some datasets/code to another datamodule. That 10k unit was and is a mess. You really should consider refactoring out some code to other units. My module has since D2005 / separation grown even worse, but it still compiles in D2007. So my answer is a) refactor and b) upgrade to D2009.
It seems clear that this is due to a slight difference between the background compiler and the real thing. You could look around (QualityCentral) what's known on that topic.
Also, since you didn't explicitly state this, you should remove unnecessary units and move uses statements down to implementation if possible. Maybe your tool could help with this.
And just to be sure you should check the unit aliases and Path settings.
You write that a full build does always succeed, but shortly after the incremental build fails with this error. Assuming that you experience this in the IDE, have you tried to use the command line compiler dcc32 to do incremental builds?
If you don't feed it the "-Q" switch (which probably most Makefiles or scripts for command line builds do) it will output a lot of information what files it compiles in what order. You could either try to do an incremental build after the error appeared in the IDE, or you could keep a command line open next to the IDE and Alt+Tab to it for compilation, skipping compilation in the IDE completely.
I simply assume you have a way to build using dcc32, one way or another - with the size of your project I can't imagine otherwise.
We regularly fall in similar problems, and we never managed (or bothered long enough) to find the precise cause. There seems to be a problem in the order which Delphi chooses to compile the units when hitting Ctrl-F9, which is incompatible with the actual dependency order of the units.
Have you tried deleting "MyBigFatUnit.dcu" before hitting Ctrl-F9?
Have you tried to re-order the declaration of your units in your dpr/dpk files, so that units appear in a correct compilation order? (i.e.: if unit B depends on unit A, unit A should appear first in the dpr/dpk)
Do you have any other projects that use part of the same codebase? If you compile one of them under different compiler settings or IFDEFs, it might change certain things in some of the DCUs which would lead to a circular dependency. A full build rebuilds all DCUs and then the problem goes away.
Try Icarus (free) from Peganza. If that does not tell you what the problem is, try their Pascal Analyzer.
We have this problem as well, also with a fairly large codebase.
We are currently using D2009, but have had this problem with all previous versions of Delphi.
It most frequently happens immediately after doing an update from source control, so I suspect there is some timestamp issue within the Delphi build process.
In our case, if Ctrl-F9 fails and reports the circular reference, a second Ctrl-F9 will generally work
A way I have been told to deal with this is to open another arbitrary file in the project, change that file, save it, and then try running the incremental compile again. Surprisingly enough, this usually works.
We have a 4 MLOC project where this comes up from time to time and this "solution" works for me.
I've fought this before, in my experience the error is quasi-legitimate. It's been a quite a while (the error has nothing to do with the version) but my memory of the situation is that it involves a loop in which part of the loop is in the implementation.
Unit A uses B in the implementation. Unit B uses A in the interface. If you compile B first it calls for A but since the call for B is in the implementation it succeeds. If you compile A first it calls for B, B turns around and calls for A in the interface, boom. Such loops are only safe if both cross references are in the implementation.
The solution is to design things so there is a minimum of stuff used in the interface and to make certain there's nothing resembling a loop in those units. So long as you keep your type definitions separate from units with code this is pretty easy to do.
The error coming and going depending on what you are doing is a hallmark of this issue as it comes down to how you enter the loop. When you do a full build the order is consistent and you either get it 100% or 0%, it's not random.
My problem is to create an ant target for automating our installer running in console mode.
The installer is created using InstallAnywhere 2008, which UniversalExtractor recognizes as a 7-zip archive. Once I have the archive unpacked, it appears that the task can use an input file to drive the console (at the very least, it appears that emitting a quit shuts everything down correctly, and output is captured).
So it looks to me as though I have all of the pieces I need for proving out this idea except a clean way to perform-self-extraction-then-stop. Searching for a command-line argument to stop the auto execution has not produced a likely candidate, and the only suitable ant task I've found ( http://www.pharmasoft.be/7z/ ) isn't so clearly documented that I have a lot of confidence in it.
The completed completed is expected to work in Windows, Linux, and a small handful of other Unix environments.
What's the best practice to use here?
Since you control the installer creation, can you run the self-extraction step on your machine, package the results before the installer is launched in a ZIP file, etc. and use that instead of the single file executable? Not very elegant but it may work.
Also, I am a bit hesitant to blatantly promote my project :) but since it has been a while since you asked the question and nobody has answered, have you considered an alternative? Our project InstallBuilder allows you to install in unattended mode directly, without having to autoextract the contents. Just invoke the executable with --mode unattended, pass any additional options you may need from the command line or an external file and you are good to go. We have a lot of ex-InstallAnywhere customers :)
I'm using P4COM to communicate with our perforce server. I have written an little utility to simplify our QA of what files have changed from one release to another. I have been using the P4COM interface from Delphi. So far so good.
I though it might be nice to allow users to view the diff between the two versions of the file from within my little utility rather than going back to p4v. So I print (get) the files at each revision using p4COM and the following command
print -o "E:\Development\TempProjects\p4Changes\temp\File_dispatch.pas#25" "//depot/mydepotpath/File_dispatch.pas"#25
and
print -o "E:\Development\TempProjects\p4Changes\temp\File_dispatch.pas#26" "//depot/mydepotpath/File_dispatch.pas"#26
However when I do this from my app using P4COM I seem to get random files (and they appear to be deleted ones). If I run the exact same command from the command line I get perfect results. Running both of these does return a file and correctly dumps it to disk where I want it, its just not the file I've asked for.
Any ideas?
Could it be a backslash issue in the command string? This would work fine at the command line, but a single backslash may be being interpreted as an escape code by whatever language compiler you are using (if C or C++, then this will definitely cause a problem, and that maybe happening under the hood on the P4COM side).
Try using double backslashes and see if that fixes it.
You're probably better of asking this to Perforce support itself, as this sounds like a bug in their software.
As a sidenote : Why do you use p4v? (I hugely prefer p4win myself)