I am seeing some very slow cache reads in my rails app. Both redis (redis-rails) and memcached (dalli) produced the same results.
It looks like it is only the first call to Rails.cache that causes the slowness (averaging 500ms).
I am using skylight to instrument my app and see a graph like:
I have a Rails.cache.fetch call in this code, but when I benchmark it I see it average around 8ms, which matches what memcache-top shows for my average call time.
I thought this might be dalli connections opening slowly, but benchmarking that didnt show anything slow either. I'm at a loss for what else to check into.
Does anyone have any good techniques for tracking this sort of thing down in a rails app?
Edit #1
Memcache server is stored in ENV['MEMCACHE_SERVERS'], all the servers are in the us-east-1 datacenter.
Cache config looks like:
config.cache_store = :dalli_store, nil, { expires_in: 1.day, compress: true }
I ran something like:
100000.times { Rails.cache.fetch('something') }
and calculated the average timings and got something on the order of 8ms when running on one of my webservers.
Testing my theory of the first request is slow, I opened a console on my web server and ran the following as the first command.
irb(main):002:0> Benchmark.ms { Rails.cache.fetch('someth') { 1 } }
Dalli::Server#connect my-cache.begfpc.0001.use1.cache.amazonaws.com:11211
=> 12.043342
Edit #2
Ok, I split out the fetch into a read and write, and tracked them independently with statsd. It looks like the averages sit around what I would expect, but the max times on the read are very spiky and get up into the 500ms range.
http://s16.postimg.org/5xlmihs79/Screen_Shot_2014_12_19_at_6_51_16_PM.png
With asana api i just want to get list of tasks completed in the last last 5 minutes. For that did below php code. I'm getting list of tasks that are not even closed.
//unix time before 5 minutes
$lastHour = time() - 5 * 60;
//convert to tz format
$last_run = date("Y-m-d\TH:i:s.000\Z", $lastHour);
$args = array('completed_since' => $last_run);
$tasks = $asana->getProjectTasks($project->id, $args);
echo "<pre>"; print_r($tasks );
$tasksJson = json_decode($tasks);
Is there anything wrong in the query?
From the docs:
Only return tasks that are either incomplete, or completed since the given time.
So completed_since always returns all incomplete tasks. Originally, this was done to model the behavior in the application: you typically saw "recently completed and all incomplete" tasks in one view. The application no longer does this, but the API still follows this convention because we can't change it and break existing clients. (Also, this is used to fetch all incomplete tasks with ?completed_since=now).
If you only want the completed tasks you can make the request you're currently using, and then filter out the ones where completed is true in PHP. It's not ideal, but I hope that clears things up!
(BTW we are working on a new, more intuitive and expressive filtering system for the API, but it's still early in the design phase and won't be shipping. So, I wouldn't hold my breath, but we are aware that this situation needs fixing ;-))
I have an XQuery query intended to wipe test documents from the database before each test that is run. Essentially it looks for a certain element to be present as a top level element in a document (called 'forTestOnly') and if it finds it it deletes the document. This query is run before each test in order to ensure the tests don't interfere with one another (we have about 200 tests using this). The exact XQuery is as such:
xquery version "1.0-ml";
import module namespace dls = "http://marklogic.com/xdmp/dls" at "/MarkLogic/dls.xqy";
let $deleteNonManagedDocs := for $testDoc in /*[forTestOnly]
let $testDocUri := fn:base-uri($testDoc)
where fn:not(dls:document-is-managed($testDocUri))
return xdmp:document-delete($testDocUri)
let $deleteManagedDocs := for $testDoc in cts:search(/*[forTestOnly], dls:documents-query())
let $testDocUri := fn:base-uri($testDoc)
return dls:document-delete($testDocUri, fn:false(), fn:false())
return ($deleteManagedDocs, $deleteNonManagedDocs)
While it seems to work fine most of the time, it recently has begun to sporadically spiral out of control. At some point during the test execution it begins to run for a near indefinite amount of time (I usually stop it after 600-700 seconds), most of the time it takes less than a second though. The database used for testing is not large (it has a few basic seed documents but nothing compared to a production database), and typically each test only creates a handful of documents with the 'forTestOnly' (if not less).
The query seems simple enough, and although running it 200 times in relatively quick succession would understandably put a strain on the database I can't imagine it would cause this kind of lagging (the tests are Grails integration tests and the entire execution takes a little over two minutes). Any ideas why the long run time?
As a side note I have verified that when the tests stall it is indeed after the XQuery has begun to run and not before in some sort of test wiring/execution.
Any help is greatly appreciated.
The query might look simple, but it isn't necessarily simple to evaluate. Those dls function calls could be doing anything, so it's tricky to estimate the complexity. The use of DLS also means that we don't know how much version history has to be deleted to delete each document.
One possibility is that you've discovered a bug. It might already be fixed, which is a good reason why you should always report the full version of the software you're using. The answer may be as simple as upgrading to pick up the fix.
Another possibility is that your test suite ends up running all of this work in a single high-level evaluation, so everything's in memory until the end. That could use enough memory to drive the server into swap. That would explain the recent "spiral out of control" behavior. Check the OS and see what it says.
Next, set the group file-log-level=Debug and check ErrorLog.txt while one of these slow events is happening. If you see XDMP-DEADLOCK messages, you may have a problem where two or more copies of this delete query are running at the same time. MarkLogic has automatic deadlock detection and resolution, but it's faster to avoid the deadlock in the first place.
Some logging might also help determine where the time is spent. Something like:
let $deleteNonManagedDocs := for $testDoc in /*[forTestOnly]
let $testDocUri := fn:base-uri($testDoc)
where fn:not(dls:document-is-managed($testDocUri))
return (
xdmp:log(text { 'unmanaged', $testDocUri }),
xdmp:document-delete($testDocUri))
let $deleteManagedDocs := for $testDoc in cts:search(/*[forTestOnly], dls:documents-query())
let $testDocUri := fn:base-uri($testDoc)
let $_ := xdmp:log(text { 'managed', $testDocUri })
return dls:document-delete($testDocUri, fn:false(), fn:false())
return ()
Finally you should also be able to simplify the query a bit. Since you're deleting everything, you can just ignore DLS.
xdmp:document-delete(
cts:uris(
(), (),
cts:element-query(xs:QName('forTestOnly'), cts:and-query(())))
This would be even simpler and more efficient if you set a collection on every test document: xdmp:collection-delete('test-docs').
I want to load a text file in Session.
The file size is about 50KB ~ 100KB.
When user trigger the function in my page. it will create the Session.
My Server's RAM is about 8GB. and the max users is about 100
Because there will be a script run in background to collect IP and MAC in LAN.
The script continues write data into text file.
In the same time, the webpage will using Ajax to fetch fresh data from text file.and display on the page.
Is it suitable to implement by session to keep the result? or any better way to achieve ?
Thanks ~
The Python script will collect the data in the LAN in 1 ~ 3 minutes.(Background job)
To avoid blocking for 1~3 minutes. I will use Ajax to fetch the data in text file (continuing added by Python script) and show on the page.
And my user should carry the information cross pages. So I want to store the data in Session.
00:02:D1:19:AA:50: 172.19.13.39
00:02:D1:13:E8:10: 172.19.12.40
00:02:D1:13:EB:06: 172.19.1.83
C8:9C:DC:6F:41:CD: 172.19.12.73
C8:9C:DC:A4:FC:07: 172.19.12.21
00:02:D1:19:9B:72: 172.19.13.130
00:02:D1:13:EB:04: 172.19.13.40
00:02:D1:15:E1:58: 172.19.12.37
00:02:D1:22:7A:4D: 172.19.11.84
00:02:D1:24:E7:0F: 172.19.1.79
00:FD:83:71:00:10: 172.19.11.45
00:02:D1:24:E7:0D: 172.19.1.77
00:02:D1:81:00:02: 172.19.11.58
00:02:D1:24:36:35: 172.19.11.226
00:02:D1:1E:18:CA: 172.19.12.45
00:02:D1:0D:C5:A8: 172.19.1.45
74:27:EA:29:80:3E: 172.19.12.62
Why does this need to be stored in the browser? Couldn't you fire off what you're collecting to a data store somewhere?
Anyway, assuming you HAD to do this, and the example you gave is pretty close to the data you'll actually be seeing, you have a lot of redundant data there. You could save space for the IPs by creating a hash pointing to each successive value, I.E.
{172 => {19 => {13 => [39], 12 => [40, 73, 21], 1 => [83]}}} ...etc. Similarly for the MAC addresses. But again, you can probably simplify this problem a LOT by storing the info you need somewhere other than the session.
I'm using jQuery-ui's autocomplete on a search form. In development the request hits my index page which returns a JSON response:
The response looks like so:
[{"id":0,"listing_id":0,"category_id":0,"title":"Natural Woven Linen
Ring Sling","description":"This natural woven linen sling is perfect
for keeping you and baby comfortable in any climate. With light
comfort, it will conform to you and your child\u0026#39;s body and
become softer over time.\nSlings are a great way to keep your baby
close and your hands free.\nWearing your baby increases bonding by
encouraging skin to skin contact and closeness with parents and
caregivers. Baby slings mimic the womb environment, making baby feel
safe and secure. Baby\u0026#39;s needs are easily met when held close,
which means less crying. Baby slings are great for discreet
breastfeeding no matter where you are. \nThis sling is suitable for
babies between 7-35 Pounds.\nBe sure to exercise caution when wearing
your baby.\nKeep baby\u0026#39;s face visible at all
times.\nPractice wearing your sling before putting baby
inside.\nAvoid any unsafe activities while wearing your baby, such
as:\nSmoking, drinking hot drinks, running, exercising, cooking, or
drinking
alcohol.","price":"65.00","currency_code":"CAD","quantity":1,"tags":["sling
rings","ring sling","toddler","newborn","natural linen","woven
wrap","baby carrier","woven sling","babywearing","baby wrap","baby
sling"],"category_path":["Bags and
Purses"],"taxonomy_path":["Accessories","Baby Accessories","Baby
Carriers \u0026 Wraps"],"materials":["aluminum rings","european
linen","cotton
thread"],"featured_rank":null,"url":"https://www.etsy.com/listing/272579256/natural-woven-linen-ring-sling?utm_source=etsyinventorymerger\u0026utm_medium=api\u0026utm_campaign=api","views":19,"num_favorers":0,"shipping_template_id":6281647,"shipping_profile_id":null,"images":["https://img1.etsystatic.com/135/0/6276910/il_170x135.987731269_rwab.jpg","https://img1.etsystatic.com/139/0/6276910/il_170x135.987731277_1q29.jpg","https://img1.etsystatic.com/140/0/6276910/il_170x135.987731279_q5lv.jpg"],"created_at":"2016-03-28T20:01:41.722Z","updated_at":"2016-03-28T20:04:52.721Z"},{"id":18,"listing_id":269532744,"category_id":269532744,"title":"Woven
Cotton Whimsical Waves Ring Sling","description":"This sling is
lightweight, yet sturdy and made from 100% cotton. The shoulder is
sewn to comfortably keep your arms free and baby\u0026#39;s weight
evenly distributed. Great for any climate and perfect for any outfit.
\n\nSlings are a great way to keep your baby close and your hands
free.\nWearing your baby increases bonding by encouraging skin to skin
contact and closeness with parents and caregivers. Baby slings mimic
the womb environment, making baby feel safe and secure.
Baby\u0026#39;s needs are easily met when held close, which means less
crying. Baby slings are great for discreet breastfeeding no matter
where you are. \nThis sling is suitable for babies between 7-35
Pounds.\nBe sure to exercise caution when wearing your baby.\nKeep
baby\u0026#39;s face visible at all times.\nPractice wearing your
sling before putting baby inside.\nAvoid any unsafe activities while
wearing your baby, such as:\nSmoking, drinking hot drinks, running,
exercising, cooking, or drinking
alcohol.","price":"65.00","currency_code":"CAD","quantity":1,"tags":["new
mom gift","baby shower gift","newborn sling","baby wrap","ring sling
tail","woven cotton sling","woven ring sling","baby carrier","ring
sling","canadian made"],"category_path":["Bags and
Purses"],"taxonomy_path":["Accessories","Baby Accessories","Baby
Carriers \u0026 Wraps"],"materials":["cotton","aluminum rings","cotton
thread"],"featured_rank":1,"url":"https://www.etsy.com/listing/269532744/woven-cotton-whimsical-waves-ring-sling?utm_source=etsyinventorymerger\u0026utm_medium=api\u0026utm_campaign=api","views":42,"num_favorers":3,"shipping_template_id":6281647,"shipping_profile_id":null,"images":["https://img1.etsystatic.com/115/0/6276910/il_170x135.927557949_lp3o.jpg","https://img1.etsystatic.com/113/0/6276910/il_170x135.927557945_8km2.jpg","https://img1.etsystatic.com/117/0/6276910/il_170x135.927557953_nyef.jpg","https://img0.etsystatic.com/112/0/6276910/il_170x135.927814742_9wo0.jpg","https://img1.etsystatic.com/127/0/6276910/il_170x135.927557973_223q.jpg"],"created_at":"2016-03-28T20:01:45.104Z","updated_at":"2016-03-28T20:04:56.129Z"}]
This is my controller:
def index
respond_to do |format|
format.html
format.json do
#etsy_products = EtsyProduct.search(params[:term])
render json: #etsy_products, status: :ok, message: 'Success'
end
end
end
My script receives the return response and formats it accordingly:
$ ->
$('#etsy_products_search').autocomplete(
minLength: 0
source: '/'
focus: (event, ui) ->
$('#etsy_products_search').val ui.item.title
false
select: (event, ui) ->
$('#etsy_products_search').val ui.item.title
$('#etsy_products_search-description').html ui.item.description
false
).autocomplete('instance')._renderItem = (ul, item) ->
$('<li>')
.attr({'title': item.description, 'data-toggle': 'tooltip', 'data-thumbnail': item.images[0], 'data-etsy-url': item.url})
.append(item.title).appendTo ul
My development log shows this:
Started GET "/?term=woven" for ::1 at 2016-03-28 17:16:49 -0400
Processing by HomeController#index as JSON
Parameters: {"term"=>"woven"}
EtsyProduct Load (0.8ms) SELECT "etsy_products".* FROM "etsy_products" WHERE (title ILIKE '%woven%')
Completed 200 OK in 18ms (Views: 13.6ms | ActiveRecord: 2.5ms)
I deployed by app, then tested it in production. The JSON response is empty, and my production log looks like so:
I, [2016-03-28T17:19:13.552941 #25285] INFO -- : Started GET "/?term=woven" for (ip) at 2016-03-28 17:19:13 -0400
I, [2016-03-28T17:19:13.558963 #25285] INFO -- : Processing by HomeController#index as JSON
I, [2016-03-28T17:19:13.559220 #25285] INFO -- : Parameters: {"term"=>"woven"}
D, [2016-03-28T17:19:13.565312 #25285] DEBUG -- : EtsyProduct Load (1.0ms) SELECT "etsy_products".* FROM "etsy_products" WHERE (title ILIKE '%woven%')
I, [2016-03-28T17:19:13.566088 #25285] INFO -- : Completed 200 OK in 7ms (Views: 2.0ms | ActiveRecord: 1.0ms)
If I go into console on the Production server, and run my search script it returns all the entries as it should. The trouble seems to be when it passes the response back to the request. It comes back empty. I'm sure it's something stupid that I've overlooked, but I can't seem to find the right answer on Google, or, (which is more likely) I'm asking the wrong question.
Turns out I was completely on the wrong track. The issue wasn't with the response, but actually with the database. You see, when I checked to ensure the database had records, I was doing the standard "rails c" method on the production server. I'm using a rake task to collect records from an API and write them to the database. So long story short, the rake task was actually writing to a development database that was on my server, and not to the production. When I ran "rails c" it was giving me access to that database and not to the production database. Urgh! Still a lot to learn. Hopefully this will help another newb in the future in case they get walled like I did.