How do I measure the amount of memory used by an executable which I run through the os/exec package in Golang? Is it better to do this through the OS itself?
You need to do this through the OS itself. If you are on plan9 or posix, Go will return the usage values from the OS for you in the structure returned by ProcessState.SysUsage().
cmd := exec.Command("command", "arg1", "arg2")
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
// check this type assertion to avoid a panic
fmt.Println("MaxRSS:", cmd.ProcessState.SysUsage().(*syscall.Rusage).Maxrss)
Note: different platforms may return this in bytes or kilobytes. Check man getrusage for details.
Related
I'm trying to pipe the output(logs) of a program to a Go program which aggregates/compress the output and uploads to S3. The command to run the program is "/program1 | /logShipper". The logShipper is written in Go and it's simply read from os.Stdin and write to a local file. The local file will be processed by another goroutine and upload to S3 periodically. There are some existing docker log drivers but we are running the container on a fully managed provider and the log processing charge is pretty expensive, so we want to bypass the existing solution and just upload to S3.
The main logic of the logShipper is simply read from the os.Stdin and write to some file. It's work correctly when running on the local machine but when running in docker the goroutine blocked at reader.ReadString('\n') and never return.
go func() {
reader := bufio.NewReader(os.Stdin)
mu.Lock()
output = openOrCreateOutputFile(&uploadQueue, workPath)
mu.Unlock()
for {
text, _ := reader.ReadString('\n')
now := time.Now().Format("2006-01-02T15:04:05.000000000Z")
mu.Lock()
output.file.Write([]byte(fmt.Sprintf("%s %s", now, text)))
mu.Unlock()
}
}()
I did some research online but not find why it's not working. One possibility I'm thinking is might docker redirect the stdout to somewhere so the PIPE not working the same way as it's running on a Linux box? (As looks like it can't read anything from program1) Any help or suggestion why it not working is welcome. Thanks.
Edit:
After doing more research I realized it's a bad practice to handle the logs in this way. I should more rely on the docker's log driver to handle the log aggregate and shipping. However, I'm still interested to find out why it's not read anything from the PIPE source program.
I'm not sure about the way the Docker handles output, but I suggest that you extract the file descriptor with os.Stdin.Fd() and then resort to using golang.org/x/sys/unix package as follows:
// Long way, for short one jump
// down straight to it.
//
// retrieve the file descriptor
// cast it to int, because Fd method
// returns uintptr
fd := int(os.Stdin.Fd())
// extract file descriptor flags
// it's safe to drop the error, since if it's there
// and it's not nil, you won't be able to read from
// Stdin anyway, unless it's a notice
// to try again, which mostly should not be
// the case
flags, _ := unix.FcntlInt(fd, unix.F_GETFL, 0)
// check if the nonblocking reading in enabled
nb := flags & unix.O_NONBLOCK != 0
// if this is the case, just enable it with
// unix.SetNonblock which is also a
// -- SHORT WAY HERE --
err = unix.SetNonblock(fd, true)
The difference between the long and a short way is that the long way will definitely tell you, if the problem is in the nonblocking state absence or not.
If this is not the case. Then I have no other ideas personally.
I'm running a couple of Java programs in BlueJ. I want to be able to know how much memory the program uses for a given input value. Is there any way I can do this?
1) to get the maximum amount of memory that your Java application can use Runtime:
Runtime runtime = Runtime.getRuntime();
System.out.println("max memory: " + runtime.maxMemory() / 1024);
2)to get how much memory that JVM has allocated for your application :
Runtime runtime = Runtime.getRuntime();
System.out.println("allocated memory: " + runtime.totalMemory() / 1024);
3)to get how much memory is being used by your application:
Runtime runtime = Runtime.getRuntime();
System.out.println("free memory: " + runtime.freeMemory() / 1024);
It could be operating system specific; notably the JVM manages Java allocated memory, but you might call external functions in C++ using JNI which are allocating memory too (e.g. your Java program calling OpenCV functions), and you could be interested in the virtual address space of your current process. On Linux you might use proc(5) and read and parse /proc/self/statm or /proc/self/stat or /proc/self/status. If your process has pid 1234, you could also try cat /proc/1234/maps ijn an other terminal, or use pmap(1). On other operating systems, you'll need to dive into their appropriate documenation.
If you only care about the memory directly used by the JVM, use the Runtime class as answered by Hans.
BTW, your JVM has a garbage collector, and so defining what exactly is the amount of memory used is not trivial.
Environment: Windows 10 Professional 64-Bit.
I want to build / install a go-project (twitterbeat as you can see).
C:\apps\Go_workspace\src\github.com\buehler\twitterbeat>go build
# github.com/buehler/twitterbeat/beater
beater\twitterbeat.go:62: b.Events undefined (type *beat.Beat has no field or method Events)
Here you can see line 62 of the file:
func (bt *Twitterbeat) Setup(b *beat.Beat) error {
logp.Info("Setup waitduration and api keys")
bt.events = b.Events
var err error
bt.period, err = time.ParseDuration(*bt.beatConfig.Period)
if err != nil {
return err
}
anaconda.SetConsumerKey(*bt.beatConfig.Twitter.ConsumerKey)
anaconda.SetConsumerSecret(*bt.beatConfig.Twitter.ConsumerSecret)
bt.api = anaconda.NewTwitterApi(*bt.beatConfig.Twitter.AccessKey, *bt.beatConfig.Twitter.AccessSecret)
return nil
}
I don't think that the code is wrong, because I donwloaded it directly from Github.
Because I am not on a linux / unix system (and I had problems with the proxy), i couldn't run "glide". Instead I donwloaded all dependencies by myself.
What can I do to build twitterbeat?
When you download the dependencies by hand, you need to make sure that they are the same version as in the glide.yaml file. The current version of beat.Beat in github.com/elastic/libbeat/beat/beat.go is newer than the one in the glide.yaml and doesn't have an Events field any more.
It's not your problem,but a fault of the library you are using.
As the code shows,it used github.com/elastic/beats/libbeat/beat,then we jump to the source of beat,the Beat struct is:
type Beat struct {
Name string // Beat name.
Version string // Beat version number. Defaults to the libbeat version when an implementation does not set a version.
UUID uuid.UUID // ID assigned to a Beat instance.
BT Beater // Beater implementation.
RawConfig *common.Config // Raw config that can be unpacked to get Beat specific config data.
Config BeatConfig // Common Beat configuration data.
Publisher *publisher.Publisher // Publisher
filters *filter.FilterList // Filters
}
It doesn't have Events field anymore!
You can use the old version of the library github.com/elastic/beats/libbeat/beat,or you can push a issue to the owner of github.com/buehler/twitterbeat to inform him to fix this bug.
I have a really odd and hard to diagnose issue with MSBuild / TFS. I have a solution that contains about 12 different build configurations. When running on the build server, it takes maybe 30mins to build the lot and has worked fine for weeks now but now is occasionally failing.
Most of the time, when it fails it'll be an error like this:
19:25:45.037 2>TestPlanDocument.cpp(1): fatal error C1093: API call 'GetAssemblyRefHash' failed '0x8007000e' : ErrorMessage: Not enough storage is available to complete this operation. [C:\Builds\1\ICCSim Card Test Controller\ICCSimCTC Release\src\CardTestController\CardTestController.vcxproj]
The error will sometimes happen on a different file. It won't happen for every build configuration either, it's very inconsistent and occasionally even builds all of them successfully. There's not much different between the build configurations either, mostly it's just a few string changes and of course they all build locally just fine.
The API call in question is usually GetAssemblyRefHash but not always. I don't think this is the issue, as Googling for GetAssemblyRefHash specifically brings up next to nothing. I suspect there's some kind of resource issue at play here but I'm at a loss as to what: There's plenty of HDD space (Hundreds of GB's), plenty of RAM (Machine originally had 4GB minimum allocated but was dynamic as it's a Hyper-v - it never pushed above 2.5GB. I upped this to 8GB minimum just in case and there's been no change).
I've set the build verbosity to diagnostic and it doesn't really show anything else that's helpful, just the same error.
For reference, the build server is fully up to date on all patches. It's running Windows Server 2012 R2, has TFS 2013 and VS 2013 installed, both are on Update 4.
I'm really at a loss at this point and would appreciate any help or pointers.
EDIT: Just to keep people up to date, the compile toolchain was in 32bit mode however even after switching to 64bit, the issue persists.
I think I found the source, but I still don't know the reason.
Browsing through the Microsoft Shared Source, we can find the source for GetAssemblyRefHash():
HRESULT CAsmLink::GetAssemblyRefHash(mdToken FileToken, const void** ppvHash, DWORD* pcbHash)
{
if (TypeFromToken(FileToken) != mdtAssemblyRef) {
VSFAIL( "You can only get AssemblyRef hashes for assemblies!");
return E_INVALIDARG;
}
HRESULT hr;
CAssembly *file = NULL;
if (FAILED(hr = m_pImports->GetFile( FileToken, (CFile**)&file)))
return hr;
return file->GetHash(ppvHash, pcbHash);
}
Only two places here to investigate - the call to m_pImports->GetFile(), where m_pImports is CAssembly *m_pImports;, the other is file->GetHash().
m_pImports->GetFile() is here, and is a dead end:
HRESULT CAssembly::GetFile(DWORD index, CFile** file)
{
if (!file)
return E_POINTER;
if (RidFromToken(index) < m_Files.Count()) {
if ((*file = m_Files.GetAt(RidFromToken(index))))
return S_OK;
}
return ReportError(E_INVALIDARG);
}
file->GetHash(), which is here:
HRESULT CAssembly::GetHash(const void ** ppvHash, DWORD *pcbHash)
{
ASSERT( ppvHash && pcbHash);
if (IsInMemory()) {
// We can't hash an InMemory file
*ppvHash = NULL;
*pcbHash = 0;
return S_FALSE;
}
if (!m_bDoHash || (m_cbHash && m_pbHash != NULL)) {
*ppvHash = m_pbHash;
*pcbHash = m_cbHash;
return S_OK;
}
DWORD cchSize = 0, result;
// AssemblyRefs ALWAYS use CALG_SHA1
ALG_ID alg = CALG_SHA1;
if (StrongNameHashSize( alg, &cchSize) == FALSE)
return ReportError(StrongNameErrorInfo());
if ((m_pbHash = new BYTE[cchSize]) == NULL)
return ReportError(E_OUTOFMEMORY);
m_cbHash = cchSize;
if ((result = GetHashFromAssemblyFileW(m_Path, &alg, (BYTE*)m_pbHash, cchSize, &m_cbHash)) != 0) {
delete [] m_pbHash;
m_pbHash = 0;
m_cbHash = 0;
}
*ppvHash = m_pbHash;
*pcbHash = m_cbHash;
return result == 0 ? S_OK : ReportError(HRESULT_FROM_WIN32(result));
}
We can see that about halfway down, it tries to allocate room to store the byte[] result, and when fails, returns E_OUTOFMEMORY, which is the error code you're seeing:
if ((m_pbHash = new BYTE[cchSize]) == NULL)
return ReportError(E_OUTOFMEMORY);
m_cbHash = cchSize;
There's other paths to consider, but this seems like the most obvious source. So it looks like the problem is that a plain memory allocation is failing.
What could cause this?
Lack of free physical memory pages / swap
Memory fragmentation in the process.
Inability to reserve commit space for this in the swap file
Lack of address space
At this point, my best guess would be memory fragmentation. Have you triple checked that the Microsoft CPP compiler is running in 64-bit mode? Perhaps see if you can debug the compiler (Microsoft symbol servers may be able to help you here), and set a breakpoint for that line and dump the heap when it happens.
Some specifics on diagnosing heap fragmentation - fire up sysinternal's VMMap when the compiler breaks, and look at the free list - you need three chunks at least 64 kB free to perform an allocation; less than 64 kB and it won't get used, and two 64 kB chunks are reserved.
Okay, I have an update to this! I opened a support ticket with Microsoft and have been busy working with them to figure out the issue.
They went down the same paths as outlined above and came to the same conclusion - it's not a resources issue.
To cut a long story short, Microsoft has now acknowledged that this is likely a bug in the VC++ compiler, which is almost certainly caused by a race condition (Though this is unconfirmed). There's no word on if they'll fix it in a future release.
There is a workaround by using the /MP flag at the project level to limit the number of compiler processes opened by MSBuild without disabling multiple instances entirely (Which for me was doubling build times).
To do this, go to your project properties and under Configuration Properties -> C/C++ -> Command Line, you need to specify the /MP flag and then a number to limit the number of processes.
My build server has 8 Virtual CPU's and the normal behaviour is equivelant to /MP8 but this causes the bug to sometimes appear. For me, using /MP4 seems to be enough to limit the bug without causing build times to increase too much. If you're seeing a bug similar to this, you may need to experiment with other numbers such as /MP6 or /MP2.
I want to collect Operating Systems Parameters..in my mbean so that after registering it i can see those values on JConsole..I have collected some parameters but I cant collect the values for ProcessCpuTime,SystemCpuLoad I tried it with OperatingSystemMXBean interface object but it doesn't work. Also i read on google that those method needs APIs which are not supported on Windows.So is there another way to calculate those values mathematically...Please help me
The Windows JVM implementation of java.lang.management.OperatingSystemMXBean is com.sun.management.OperatingSystem. You can reference it directly by casting the former:
import java.lang.management.*;
import com.sun.management.*;
...
OperatingSystem os = (OperatingSystem)ManagementFactory.getOperatingSystemMXBean();
There's some variability between the Java 6 and java 7 versions of this class, which can be observed with this Groovy script:
import java.lang.management.*;
os = ManagementFactory.getOperatingSystemMXBean();
println os.getClass().getName();
try { println "Process CPU Load:${os.getProcessCpuLoad()}"; } catch (e) {}
try { println "Process CPU Time:${os.getProcessCpuTime()}"; } catch (e) {}
try { println "System CPU Load:${os.getSystemCpuLoad()}"; } catch (e) {}
Java 6 Output:
com.sun.management.OperatingSystem
Process CPU Time:79810111600
Java 7 Output:
com.sun.management.OperatingSystem
Process CPU Load:-1.0
Process CPU Time:1840811800
System CPU Load:0.4902940980431365
So it's not as useful as the Linux [et.al.] variant, but you might get what you're looking for.