What is the difference between
cpu_stats and percpu_stats when using Docker remote API:
The request is :
GET /containers/(id or name)/stats
(A part of) The response is:
"cpu_stats" : {
"cpu_usage" : {
"percpu_usage" : [
8646879,
24472255,
36438778,
30657443
],
"usage_in_usermode" : 50000000,
"total_usage" : 100215355,
"usage_in_kernelmode" : 30000000
},
"system_cpu_usage" : 739306590000000,
"throttling_data" : {"periods":0,"throttled_periods":0,"throttled_time":0}
},
"precpu_stats" : {
"cpu_usage" : {
"percpu_usage" : [
8646879,
24350896,
36438778,
30657443
],
"usage_in_usermode" : 50000000,
"total_usage" : 100093996,
"usage_in_kernelmode" : 30000000
},
"system_cpu_usage" : 9492140000000,
"throttling_data" : {"periods":0,"throttled_periods":0,"throttled_time":0}
}
Example taken from Docker docs:
https://docs.docker.com/engine/reference/api/docker_remote_api_v1.24/#get-container-stats-based-on-resource-usage
When testing with a sample container, values are almost the same.
Example of an output:
#Cpu stas
{u'cpu_usage': {u'usage_in_usermode': 0, u'total_usage': 36569630, u'percpu_usage': [8618616, 3086454, 16466404, 8398156], u'usage_in_kernelmode': 20000000}, u'system_cpu_usage': 339324470000000, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}
#Per cup stats
{u'cpu_usage': {u'usage_in_usermode': 0, u'total_usage': 36569630, u'percpu_usage': [8618616, 3086454, 16466404, 8398156], u'usage_in_kernelmode': 20000000}, u'system_cpu_usage': 339320550000000, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}
I tried also to compare specific metrics in the two case for 4 containers:
#First container
359727340000000 #CPU Stats
359723390000000 #Per CPU Stats
#2
359735220000000
359731290000000
#3
359743100000000
359739170000000
#4
359750940000000
359747000000000
The values above are almost same (some differences but not huge - may be because there are some ms between each request.)
In the official documentation:
The precpu_stats is the cpu statistic of last read, which is used for
calculating the cpu usage percent. It is not the exact copy of the
“cpu_stats” field.
Not very clear for me.
Anyone could explain better ?
Related
I’m new to telegraf and influxdb, and currently looking forward to exploring telegraf, but unfortunetly, I have some difficulty getting started, I will try to explain my problem below:
Objectif: parsing JSON file using telegraf input plugin.
Input : https://wetransfer.com/downloads/0abf7c609d000a7c9300dc20ee0f565120200624164841/ab22bf ( JSON file used )
The input json file is a repetition of the same structure that starts from params and ends at it.
you find below the main part of the input file :
{
"events":[
{
"params":[
{
"name":"element_type",
"value":"Home_Menu"
},
{
"name":"element_id",
"value":""
},
{
"name":"uuid",
"value":"981CD435-E6BC-01E6-4FDC-B57B5CFA9824"
},
{
"name":"section_label",
"value":"HOME"
},
{
"name":"element_label",
"value":""
}
],
"libVersion":"4.2.5",
"context":{
"locale":"ro-RO",
"country":"RO",
"application_name":"spresso",
"application_version":"2.1.8",
"model":"iPhone11,8",
"os_version":"13.5",
"platform":"iOS",
"application_lang_market":"ro_RO",
"platform_width":"320",
"device_type":"mobile",
"platform_height":"480"
},
"date":"2020-05-31T09:38:55.087+03:00",
"ssid":"spresso",
"type":"MOBILEPAGELOAD",
"user":{
"anonymousid":"6BC6DC89-EEDA-4EB6-B6AD-A213A65941AF",
"userid":"2398839"
},
"reception_date":"2020-06-01T03:02:49.613Z",
"event_version":"v1"
}
Issue : Following the documentation, I tried to define a simple telegraf.conf file as below:
[[outputs.influxdb_v2]]
…
[[inputs.file]]
files = ["/home/mouhcine/json/file.json"]
json_name_key = "My_json"
#... Listing all the string fields in the json.(I put only these for simplicity reason).
json_string_fields = ["ssid","type","userid","name","value","country","model"]
data_format = "json"
json_query= "events"
Basically declaring string fields in the telegraf.conf file would do it, but I couldn’t get all the fields that are subset in the json file, like for example what’s inside ( params or context ).
So finally, I get to parse fields with the same level of hierarchy as ssid, type, libVersion, but not the ones inside ( params, context, user).
Output : Screen2 ( attachment ).
OUTPUT
By curiosity, I tried to test the documentation’s example, in order to verify whether I get the same expected result, and the answer is no :/, I don’t get to parse the string field in the subset of the file.
The doc’s example below:
Input :
{
"a": 5,
"b": {
"c": 6,
"my_field": "description"
},
"my_tag_1": "foo",
"name": "my_json"
}
telegraf.conf
[[outputs.influxdb_v2]]
…
[[inputs.file]]
files = ["/home/mouhcine/json/influx.json"]
json_name_key = "name"
tag_keys = ["my_tag_1"]
json_string_fields = ["my_field"]
data_format = "json"
Expected Output : my_json,my_tag_1=foo a=5,b_c=6,my_field="description"
The Result I get : "my_field" is missing.
Output: Screen 1 ( attachement ).
OUTPUT
By the way, I use the influxdb cloud 2, and I apologize for the long description of this little problem, I would appreciate some help please :), Thank you so much in advance.
I am reading the specification, but I can not understand the properties of the sampler.
This is the animation that I have
"animations" : [
{
"channels" : [
{
"sampler" : 0,
"target" : {
"node" : 0,
"path" : "translation"
}
}
],
"name" : "00001_2780.datAction",
"samplers" : [
{
"input" : 9,
"interpolation" : "CUBICSPLINE",
"output" : 10
}
]
},
{
"channels" : [
{
"sampler" : 0,
"target" : {
"node" : 1,
"path" : "translation"
}
}
],
"name" : "00002_2780.datAction",
"samplers" : [
{
"input" : 9,
"interpolation" : "CUBICSPLINE",
"output" : 11
}
]
}
],
What I can not understand is what are the values 9 and 10 for the first sample and 9 and 11 for the second
All that we have in the specification is
https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#animations
Each of the animation's samplers defines the input/output pair: a set of floating point scalar values representing linear time in seconds; and a set of vectors or scalars representing animated property.
And this makes it more unclear to me.
Is there a more detailed explanation about what input/output values are and what they represent. What will happen for example if I change the input from 9 to 99 or to 9.9 or to 0.9 or to 0. How will this change the animation?
Thanks
The numbers 9 and 10 here are glTF accessor index ID values. If you decode accessor index 9, you'll find the list of times for each of the keyframes of the animation. If you decode accessor 10, normally you would expect to find the list of values for the keyframes. But since this is CUBICSPLINE, accessor 10 will contain the in-tangent, value, and out-tangent for each keyframe.
One way to investigate glTF files like this is to use the glTF Tools extension for VSCode. You can right-click the input or output value and choose Go To Definition to get to the accessor in question, and choose Go To Definition again to decode it. (Disclaimer, I'm a contributor to glTF Tools).
I got a crash log like this:
{"timestamp":"2017-01-05 22:29:03.03 +0800","bug_type":"298","os_version":"iPhone OS 9.3.2 (13F69)"}
{
"crashReporterKey" : "6e088e6d206ce6025a40eacd5ea68779dc5ac344",
"kernel" : "Darwin Kernel Version 15.5.0: Mon Apr 18 16:44:06 PDT 2016; root:xnu-3248.50.21~4\/RELEASE_ARM64_T7000",
"product" : "iPhone7,2",
"incident" : "C106CE8A-F492-47FF-BEA9-293E6ACFE201",
"date" : "2017-01-05 22:29:03.03 +0800",
"build" : "iPhone OS 9.3.2 (13F69)",
"timeDelta" : 3377,
"memoryStatus" : {
"compressorSize" : 30099,
"pageSize" : 4096,
"compressions" : 719018,
"memoryPages" : {
"active" : 33364,
"throttled" : 0,
"fileBacked" : 24001,
"wired" : 169666,
"anonymous" : 27027,
"purgeable" : 6,
"inactive" : 15716,
"free" : 2836,
"speculative" : 1948
},
"uncompressed" : 81552,
"decompressions" : 333878
},
"largestProcess" : "MYAPP",
"processes" : [
{
"rpages" : 556,
"states" : [
"daemon",
"idle"
],
"name" : "assetsd",
"pid" : 839,
"reason" : "vm-pageshortage",
"fds" : 50,
"uuid" : "MYUUID",
"purgeable" : 0,
"cpuTime" : 0.373819,
"lifetimeMax" : 970
},
{
"rpages" : 231,
"states" : [
"daemon",
"idle"
],
"name" : "followupd",
"pid" : 851,
"reason" : "vm-pageshortage",
"fds" : 50,
"uuid" : "MYUUID",
"purgeable" : 0,
"cpuTime" : 0.04115,
"lifetimeMax" : 487
},
and a lot more...
There is no need to symbolize it since there is no call stack shown.
And I was just running my app to play video for like one hour, then this crash came out.
It is not every time but quite often.
How should I read such kind of log?
Can anyone explain me which of all the keys are important and what they mean in the big json?
#dstudeba is right, when you playing the video your CPU memory consumption is more then it is expected. Sometimes it is happened
when there is bulk of data coming from the server.
when you does not managed the memory in your app.
For more details : Analyzing Crash Reports and MemoryMgmt
Your App crashed because it ran out of memory. I don't know what your app does, but it either has a memory leak or is not monitoring/releasing memory properly.
Under reason it says [vm-pageshortage] according to the Apple documentation it is "The process was killed due to memory pressure."
You can read Apple's documentation on crash reports here: https://developer.apple.com/library/content/technotes/tn2151/_index.html
Docker provides an interactive stats command, docker stats [cid] which gives up to date information on the CPU usage, like so:
CONTAINER CPU % MEM USAGE/LIMIT MEM % NET I/O
36e8a65d 0.03% 4.086 MiB/7.798 GiB 0.05% 281.3 MiB/288.3 MiB
I'm trying to get the CPU usage as a percentage in a digestible format to do some analysis.
I've seen the stats in /sys/fs which seem to provide similar values as the Docker Remote API which gives me this JSON blob:
{
"cpu_usage": {
"usage_in_usermode": 345230000000,
"total_usage": 430576697133,
"percpu_usage": [
112999686856,
106377031910,
113291361597,
97908616770
],
"usage_in_kernelmode": 80670000000
},
"system_cpu_usage": 440576670000000,
"throttling_data": {
"throttled_time": 0,
"periods": 0,
"throttled_periods": 0
}
}
But I'm unsure how to get an exact CPU Usage as a percentage from that.
Any ideas?
If you are going to use the Stats API call - you can take a look at how the docker client does it: https://github.com/docker/docker/blob/eb131c5383db8cac633919f82abad86c99bffbe5/cli/command/container/stats_helpers.go#L175-L188
func calculateCPUPercent(previousCPU, previousSystem uint64, v *types.StatsJSON) float64 {
var (
cpuPercent = 0.0
// calculate the change for the cpu usage of the container in between readings
cpuDelta = float64(v.CPUStats.CPUUsage.TotalUsage) - float64(previousCPU)
// calculate the change for the entire system between readings
systemDelta = float64(v.CPUStats.SystemUsage) - float64(previousSystem)
)
if systemDelta > 0.0 && cpuDelta > 0.0 {
cpuPercent = (cpuDelta / systemDelta) * float64(len(v.CPUStats.CPUUsage.PercpuUsage)) * 100.0
}
return cpuPercent
}
Basically, you take a point of reference, then see the difference in say 10 secs, you can then tell how much of the time was used by the container. Say, we start with 0 SystemCPUUsage and 0 CPUUsage for the container. If after 10 secs, we have 10 SystemCPUUsage and 1 CPUUsage, then we have 10% usage. You are just given the results in nanoseconds, not seconds, in the API. The actual time does not matter, the total SystemCPUUsage change is what matters, then compare CPUUSage to that.
After we consume the remote api we get these fields: precpu_stats/cpu_stats
Then, basically here is the code: (javascript example)
var res <---- remote api response
var cpuDelta = res.cpu_stats.cpu_usage.total_usage - res.precpu_stats.cpu_usage.total_usage;
var systemDelta = res.cpu_stats.system_cpu_usage - res.precpu_stats.system_cpu_usage;
var RESULT_CPU_USAGE = cpuDelta / systemDelta * 100;
Just to clarify the RESULT_CPU_USAGE... it's the amount of resource consumed from your physical hardware, so supposing you are getting RESULT_CPU_USAGE as 50%, it means that 50% of all your PC power is being used by container X
So I need this also, and the following gives me the correct CPU usage, factoring in number of cores.
var cpuDelta = metric.cpu_stats.cpu_usage.total_usage - metric.precpu_stats.cpu_usage.total_usage;
var systemDelta = metric.cpu_stats.system_cpu_usage - metric.precpu_stats.system_cpu_usage;
var RESULT_CPU_USAGE = cpuDelta / systemDelta * metric.cpu_stats.cpu_usage.percpu_usage.length * 100;
console.log(RESULT_CPU_USAGE);
I faced with a limitation problem using tweepy. I am recieving Rate limit exceeded error every time running script. I need to know is there any way to know how many requests may I do before Rate limit exceeded error occured.
Tweepy offers access to the Rate Limit API.
From their documentation
import tweepy
consumer_key = 'a'
consumer_secret = 'b'
access_token = 'c'
access_token_secret = 'd'
# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)
# Creation of the actual interface, using authentication
api = tweepy.API(auth)
# Show the rate Limits
print api.rate_limit_status()
You'll then see a list of all the available rate limits and how many calls you have remaining.
For example:
{ "rate_limit_context" : { "access_token" : "1234" },
"resources" : { "account" : { "/account/login_verification_enrollment" : { "limit" : 15,
"remaining" : 15,
"reset" : 1411295469
},
"/account/settings" : { "limit" : 15,
"remaining" : 15,
"reset" : 1411295469
},
"/account/update_profile" : { "limit" : 15,
"remaining" : 15,
"reset" : 1411295469
},
"/account/verify_credentials" : { "limit" : 15,
"remaining" : 15,
"reset" : 1411295469
}
The rate limits can be found in the Twitter API documentation:
https://dev.twitter.com/docs/rate-limiting/1#rest