I am running appium tests on ios that are running very, very slowly on some machines, but much faster on another. Both machines have somewhat similar specs, but the performance differential is substantial. For instance, I have 4 text fields I am trying to send text to. On the faster machine, this is how fast it's running:
##### Starting get element with locator {:xpath=>"//XCUIElementTypeOther[#name='main'][1]/XCUIElementTypeOther[3]/XCUIElementTypeTextField"}
##### Time to get element: 0.6259970664978027 seconds
##### Time to send keys: 1.412217140197754 seconds
##### Starting get element with locator {:xpath=>"//XCUIElementTypeOther[#name='main'][1]/XCUIElementTypeOther[5]/XCUIElementTypeTextField"}
##### Time to get element: 0.8429489135742188 seconds
##### Time to send keys: 1.5375289916992188 seconds
##### Starting get element with locator {:xpath=>"//XCUIElementTypeOther[#name='main'][1]/XCUIElementTypeOther[7]/XCUIElementTypeTextField"}
##### Time to get element: 0.8602356910705566 seconds
##### Time to send keys: 2.2906301021575928 seconds
##### Starting get element with locator {:xpath=>"//XCUIElementTypeOther[#name='main'][1]/XCUIElementTypeOther[9]/XCUIElementTypeSecureTextField"}
##### Time to get element: 0.8182432651519775 seconds
##### Time to send keys: 1.605348825454712 seconds
^^^ Machine specs:
Macbook Pro (15-inch 2017)
Processor : 2.9 Ghz Quad-Core I7
Memory: 16 GB
Graphics : Radeon Pro 560 4GB
Intel HD Graphics 630 1536 MB
And, on the slower machine, this is the result:
##### Starting get element with locator {:xpath=>"//XCUIElementTypeOther[#name='main'][1]/XCUIElementTypeOther[3]/XCUIElementTypeTextField"}
##### Time to get element: 3.373900890350342 seconds
##### Time to send keys: 5.192986011505127 seconds
##### Starting get element with locator {:xpath=>"//XCUIElementTypeOther[#name='main'][1]/XCUIElementTypeOther[5]/XCUIElementTypeTextField"}
##### Time to get element: 4.875856876373291 seconds
##### Time to send keys: 5.553454875946045 seconds
##### Starting get element with locator {:xpath=>"//XCUIElementTypeOther[#name='main'][1]/XCUIElementTypeOther[7]/XCUIElementTypeTextField"}
##### Time to get element: 4.758421897888184 seconds
##### Time to send keys: 5.982282876968384 seconds
##### Starting get element with locator {:xpath=>"//XCUIElementTypeOther[#name='main'][1]/XCUIElementTypeOther[9]/XCUIElementTypeSecureTextField"}
##### Time to get element: 4.933738946914673 seconds
##### Time to send keys: 5.845119953155518 seconds
^^^ Machine specs:
Macbook Pro (15-inch 2016)
Processor : 2.7 Ghz Quad-Core I7
Memory: 16 GB
Graphics : Intel HD Graphics 530 1536 MB
I understand I'm using a long xpath locator, this is necessary for these elements. The issue I have is why is there a substantial performance difference between the two. Could the additional Radeon graphics card be the reason? Is there setting on the ios simulator in the slower machine I'm missing? I understand that this is an open ended question, but I've researched this issue for days with no results. Any help would be appreciated.
Since you are using different computers, I have few questions in my mind...
Are you using the same Xcode & Simulators in both computers?
What about the Appium?
Ideally with almost identical computers, you shouldn't be seeing much of a difference. Could you please share the Xcode & Appium versions.
Related
Due to temp (hopefully) financial problems I have to use old laptop. It's FSB (Front Side Bridge) clock is 333MHz (https://www.techsiting.com/mt-s-vs-mhz/). It has 2 SO-DIMM slots for DDR2 SDRAM. It had only 1 DIMM 2 Gb previously and it was a nightmare.
Each slot can handle maximum 2Gb so maximum amount of memory is 4Gb. Knowing that supported DDR stands for double data ratio, I've bought for funny money (10 euro) 2 DDR2 DIMM SO-DIMM 800MHz hoping to get (assuming memory divider is 1:2 - it's a double data ratio, isn't it?) 2x333MHz->apply divider=667MT/s (no idea how they have avoided 666). As I have Core2Duo I even had a very little hope to get 4x333MHz=1333MT/s.
But it seems that my memory divider is 1:1, so I get either
2x333MHzxDivider=333MT/s
4x333MHzxDivider=?
And utilities like lshw and dmidecode seem to confirm that:
~ >>> sudo lshw -C memory | grep clock
clock: 333MHz (3.0ns) # notice 333MHz here
clock: 333MHz (3.0ns) # notice 333MHz here
~ >>> sudo dmidecode --type memory | grep Speed
Supported Speeds:
Current Speed: Unknown
Current Speed: Unknown
Speed: 333 MT/s # notice 333MT/s here
Speed: 333 MT/s # notice 333MT/s here
~ >>>
So my 333MHz on FSB has been multiplied by 1 (one) and I've got 333MT/s (if I understood correct). I'm still satisfied: OS does not swap that much, boot process is faster, programs starts faster, browser does not hang every hour and I can open much more tabs). I just want to know, since I have Core2Duo what **MT/s8*8 do I have from these two? Or maybe it is even more comlicated?
2x333MHzxDivider=333MT/s
4x333MHzxDivider=667MT/s # 4 because of Duo
and is there any difference for 2 processors system with just 4Gb of RAM with MT\s == MHz?
PS BIOS is old (although latest) and I cannot see real FSB clock there, nor change it nor change the memory divider.
Looks like there's no point in checking I/O bus clock using some Linux command/tool because it is just always half of memory clock.
if what is written in electronics.stackexchange.com/a/424928:
I/O bus clock is always half of bus data rate.
my old machine has these parameters:
It is DDR2-333 (not standardized by JEDEC since they start from DDR-400)
It has memory MHz = 333
It has memory MT/s = 333
It has I/O bus MHz = 166.5 # just because
The thing I still don't get is that I have Core2Duo, so is my memory MT/s = 333 or 666.
I am a newbie to Glusterfs. I have currently setup Glusterfs in two servers with following options:
performance.cache-size 2GB
performance.io-thread-count 16
performance.client-io-threads on
performance.io-cache on
performance.readdir-ahead on
When I run my binary in following manner:
./binary > shared_location/log
It takes roughly 1.5 mins
log size is roughly 100M
Whereas running in this manner:
./binary > local_location/log
Takes roughly 10 secs.
This is a huge difference in time. No of cores in glusterfs server: 2, current machine: 2
Is there any way I can reduce time?
Also, is there any standard configuration to start off with, so I can avoid basic issues like above?
We have a 1 GB List that was created using View.asList() method on beam sdk 2.0. We are trying to iterate through every member of the list and do, for now, nothing significant with it (we just sum up a value). Just reading this 1 GB list is taking about 8 minutes to do so (and that was after we set the workerCacheMb=8000, which we think means that the worker cache is 8 GB). (If we don't set the workerCacheMB to 8000, it takes over 50 minutes before we just kill the job.). We're using a n1-standard-32 instance, which should have more than enough RAM. There is ONLY a single thread reading this 8GB list. We know this because we create a dummy PCollection of one integer and we use it to then read this 8GB ViewList side-input.
It should not take 6 minutes to read in a 1 GB list, especially if there's enough RAM. EVEN if the list were materialized to disk (which it shouldn't be), a normal single NON-ssd disk can read data at 100 MB/s, so it should take ~10 seconds to read in this absolute worst case scenario....
What are we doing wrong? Did we discover a dataflow bug? Or maybe the workerCachMB is really in KB instead of MB? We're tearing our hair out here....
Try to use setWorkervacheMb(1000). 1000 MB = Around 1GB. It will pick the side input from cache of each worker node and that will be fast.
DataflowWorkerHarnessOptions options = PipelineOptionsFactory.create().cloneAs(DataflowWorkerHarnessOptions.class);
options.setWorkerCacheMb(1000);
Is it really required to iterate 1 GB of side input data every time or you are need some specific data to get during iteration?
In case you need specific data then you should get it by passing specific index in the list. Getting data specific to index is much faster operation then iterating whole 1GB data.
After checking with the Dataflow team, the rate of 1GB in 8 minutes sounds about right.
Side inputs in Dataflow are always serialized. This is because to have a side input, a view of a PCollection must be generated. Dataflow does this by serializing it to a special indexed file.
If you give more information about your use case, we can help you think of ways of doing it in a faster manner.
Simple question:
What is the memory limitation of the Pig LOAD statement?
More detailed question:
Is there any way to relate available physical resources (disk, RAM, CPU) to the maximum size of a directory that a Pig LOAD statement can handle?
Scenario:
A research project is using a Pig script that is trying to load a directory containing 12,000+ files with a total size of 891GB in a single Pig LOAD statement, copied below. The files are gzipped WAT files which describe, in raw text, a collection of web pages. When run, the job appears to crash/hang/freeze our cluster every time. Since we are all new to Hadoop, the suspicion has been on resources and configuration until I finally was able to review the code.
-- load data from I_WATS_DIR
Orig = LOAD '$I_WATS_DIR' USING org.archive.hadoop.ArchiveJSONViewLoader('Envelope.ARC-Header- Metadata.Target-URI','var2...','var3...','var4...{1,2,3,4} as
(src:chararray,timestamp:chararray,html_base:chararray,relative:chararray,path:chararray,text:chararray,alt:chararray);
Details:
CLUSTER
1 front end node, 16 cores, 64GB RAM, 128GB swap, NameNode
3 compute nodes, 16 cores, 128GB RAM, 128GB swap, DataNode
TEST JOB 1
Same script referenced above, loading a directory with 1 file
Resident memory reported 1.2GB
Input: 138MB
Output: 207MB
Reduce input records: 1,630,477
Duration: 4m 11s
TEST JOB 2
Same script, 17 files
Resident memory: 16.4GB
Input: 3.5GB
Output: 1.3GB
Reduce input records: 10,648,807
Duration: 6m 48s
TEST JOB 3
Same script, 51 files
Resident memory: 41.4GB
Input: 10.9GB
Output: not recorded
Reduce input records: 31,968,331
Duration: 6m 18s
Final thoughts:
This is a 4 node cluster with nothing else running on it, fully dedicated to Cloudera Hadoop CDH4, running this 1 job only. Hoping this is all the info people need to answer my original question! I strongly suspect that some sort of file parsing loop that loads 1 file at a time is the solution, but I know even less about Pig than I do about Hadoop. I do have a programming/development background, but in this case I am the sys admin, not the researcher or programmer.
Based on your description of your cluster and the amount of data your pushing through it, it sounds like you are running out of space during the map/shuffle phase of the job. The temporary data is sent over the network, uncompressed, and then written to disk on the reducer before being processed in the reduce phase. One thing you can try is to compress the output of the mappers by setting mapred.map.compress.output to true (and specifying the desired codec).
But with only four nodes, I suspect you're just trying to do too much at once. If you can, try splitting up your job into multiple steps. For example, if you are doing the standard word count example, do the word count over small portions of your data, and then run a second MR program that sums those counts.
I'd like to view historical data for guest cpu/memory/IO usage, rather than just current usage.
There is a perl program i have written that does this. See link text
It also supports logging to a URL.
Features:
perl xenstat.pl -- generate cpu stats every 5 secs
perl xenstat.pl 10 -- generate cpu stats every 10 secs
perl xenstat.pl 5 2 -- generate cpu stats every 5 secs, 2 samples
perl xenstat.pl d 3 -- generate disk stats every 3 secs
perl xenstat.pl n 3 -- generate network stats every 3 secs
perl xenstat.pl a 5 -- generate cpu avail (e.g. cpu idle) stats every 5 secs
perl xenstat.pl 3 1 http://server/log.php -- gather 3 secs cpu stats and send to URL
perl xenstat.pl d 4 1 http://server/log.php -- gather 4 secs disk stats and send to URL
perl xenstat.pl n 5 1 http://server/log.php -- gather 5 secs network stats and send to URL
Sample output:
[server~]# xenstat 5
cpus=2
40_falcon 2.67% 2.51 cpu hrs in 1.96 days ( 2 vcpu, 2048 M)
52_python 0.24% 747.57 cpu secs in 1.79 days ( 2 vcpu, 1500 M)
54_garuda_0 0.44% 2252.32 cpu secs in 2.96 days ( 2 vcpu, 750 M)
Dom-0 2.24% 9.24 cpu hrs in 8.59 days ( 2 vcpu, 564 M)
40_falc 52_pyth 54_garu Dom-0 Idle
2009-10-02 19:31:20 0.1 0.1 82.5 17.3 0.0 *****
2009-10-02 19:31:25 0.1 0.1 64.0 9.3 26.5 ****
2009-10-02 19:31:30 0.1 0.0 50.0 49.9 0.0 *****
Try Nagios, or Munin.
Xentop is a tool to monitor the domains (VMs) running under Xen. VMware's ESX has a similar tool (I believe its called esxtop).
The problem is that you'd like to see the historical CPU/Mem usage for domains on your Xen system, correct?
As with all Virtualization layers, there are two views of this information relevant to admins: the burden imposed by the domain on the host and the what the domain thinks is its process load. If the domain thinks it is running low on resources but the host is not, it is easy to allocate more resources to the domain from the host. If the host runs out of resources, you'll need to optimize or turn off some of the domains.
Unfortunately, I don't know of any free tools to do this. XenSource provides a rich XML-RPC API to control and monitor their systems. You could easily build something from that.
If you only care about the domain-view of its own resources, I'm sure there are plenty of monitoring tools already available that fit your need.
As a disclaimer, I should mention that the company I work for, Leostream, builds virtualization management software. Unfortunately, it does not really do utilization monitoring.
Hope this helps.
Both Nagios and Munin seem to have plugins/support for Xen data collection.
A Xen Virtual Machine Monitor Plugin for Nagios
munin plugins