I have a ubuntu 18.04 system with 4 gb ram and 500gb hard disk.I have installed elastic search in my system like "sudo apt-get install elasticsearch"
but while starting elastic search with command : sudo service elasticsearch start
My system getting stuck, I am unable to do any thing. how to fix this
Maybe I can answer this question as I have encountered this problem a while ago and have also resolved it after some R&D.My ROR application is Live and have more than 8000+ images and CPU Intensive as there is lot more happening in my server as every request uses LAT/LNG to serve a geo-calculated response.I was facing this low memory issue similarly like yours when i started and reindexed my data on elasticsearch.
My Learnings-
ElasticSearch is a monster that will eat up your ram.
No Matter how optimized your keywords are OR the data is, that you are pushing to the elasticSearch to get indexed, you must keep a default buffer of backup memory.
What I did...that resolved my issue...
Add a Swap space which acts like a backup memory in case your RAM is eaten up.
I also upgraded to 8GB HD to be on safer side.
Now since i have followed the above mentioned steps, i am not facing any low memory error despite the size of my indexex data becoming huge every month.Furthermore I am still good to go error free and memory related issues while indexing this huge data.
One more thing, which I recently did to improve memory consumption of my Rails app is injecting jmalloc inside my ruby 2.4.1.You can read more here but in simple words, It helps improve memory consumption of ruby apps.A copied explanation of Jmalloc would be like -
Ruby traditionally uses the C language function malloc to dynamically allocate, release, and re-allocate memory when storing objects. Jemalloc is a malloc(3) implementation developed by Jason Evans (hence the “je” initials at the start of malloc), which appears to be more effective at allocating memory compared to other allocators due to its focus on fragmentation avoidance and scalable concurrency support.
Below are my steps that I took to add Jmalloc into my existing Ruby 2.4.1 by reinstalling 2.4.1 on my Production server(after testing on staging/dev) using rvm.
===========CHECK RVM --
rvm info
============ check ruby version
ruby -v
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
========== if ruby is installed, reinstall with Jemalloc and compile
=============REINSTALL RUBY WITH JAMALLOC
rvm reinstall 2.4.1 -C --with-jemalloc --disable-binary
============VERIFY JMALLOC IN THE BELOW LIST of ruby compiling list
ruby -r rbconfig -e "puts RbConfig::CONFIG['LIBS']"
-lpthread -ljemalloc -lgmp -ldl -lcrypt -lm
The Results of jmalloc being injected in ruby were just outstanding even after testing with 2000 random requests, my overall App memory was only 364MB and remained the same throughout by testing with below mentioned testing gems.
gem "memory_profiler"
gem "derailed_benchmarks"
hope it helps
CPU and (RAM)memory used by Elasticsearch is very high. So you can tell the elastic search that you can only use a specific amount of memory.
First go to /etc/elasticsearch/
Just see the directory structure and you will get a file named jvm.options
Just allocate the memory by adding 2 lines of code in jvm.options file:
-Xms1g -Xmx1g
Just restart your elasticsearch and enjoy it :)
Related
How can we recover a dump from a database version 3.0.3 to the actual version 4.1.3 Community?
I've searched and tried many different alternatives, but no one have still worked.
In one of this alternatives we've tried the command
neo4j-admin backup --backup-dir=/home/ec2-user --verbose,
but the commands to recover are not working, for example:
neo4j-admin restore --from=/var/lib/neo4j/data/databases/neo4j --database=system --force
We are getting the error message:
WARNING: Max 1024 open files allowed, minimum of 40000 recommended. See the Neo4j manual.
Unmatched arguments from index 0: 'restore', '--from=/var/lib/neo4j/data/databases/neo4j', '--verbose', '--database=neo4j', '--force'
Did you mean: store-info or report or memrec?
This seems like a reasonable (but time-consuming) approach:
If you do not have neo4j 3.0.3 installed, get it from here (as documented here), and install it.
Follow the simple 3.0 instructions for restoring a backup (to your 3.0.3 installation). The 3.0 restoration process is very different from the 4.x process.
Upgrade the 3.0.3 installation to 4.1.3, stepwise, by:
upgrading to 3.3, then
upgrading to 3.5.22, then
migrating to 4.x
Good luck.
The community edition of neo4j doesn't have the restore command, only the enterprise edition does. You can take a look at the table here.
On my old machine I can run middleman. I'm now trying to run my site on my new machine (bundle exec middleman), and I'm having issues, running into the warning/error: circular argument reference.
I'm no longer in contact with the person (ex) who set up the app so I'm doing my best to sort through this issue based on my very limited rails knowledge. I've looked up this issue but none of the answers I'm seeing seem to be working.
Enviroment Info
macOS high sierra version 10.13
ruby 2.3.3p222
middleman v4
I keep getting the same type of error messages and I have really done all I can to get rid of them. The message is as follows, only the module can vary at times:
ruby 5740 child_info_fork::abort: address space needed by 'date_core.so' (0x600000) is already occupied
These messages occur at various times when trying to run Rails, such as “rails console”.
I am running the following environment:
Windows 7 Ultimate 64-bit
Cygwin Net using setup 2.774 (installed from Internet mirrors)
Ruby 1.9.3p392
Rails 3.2.11
I have taken the following steps attempting to fix this:
Deleted all BLODA and restarted, including Norton 360 and Malwarebytes. (BLODA is Big List of Dodgy Apps.)
Deleted and reinstalled Cygwin.
Executed “rvm implode” and rebuilt the RVM and Ruby on Rails environment.
Used CMD & ASH to run “peflagsall” & “rebaseall”. Ran “rubyrebase”. Restarted.
Used FIND to build a file of all *.so & *.dll files. Ran “rebase -T”. Restarted. This has also been done using the “-b 0x77000000” switch.
Used same file to create “to_peflags” script issuing peflags against each individual file. Restarted.
All repair attempts have been futile. At times, “rebase -si |grep ‘*’” would be clean. Any attempt to run “rails console” or similar commands results in Address Space Occupied messages. Subsequent “rebase -si” would then again show conflicts.
I have been trying to run “peflags” against the Ruby modules, but I only get “skipped because could not open” error messages there. File permissions and ACL both show correct permissions. The same error occurs whether or not Administrator is used.
This is pretty frustrating. I would like to fix my current environment, but I am considering moving to MSYS or SFU, but I don’t know if either of those are decent options. I am also considering going native UNIX, but I’ll leap off the cliff when I get to it.
For now, does anyone have a fix for this specific error for me? Thanks…
I have used procexp to monitor "rails console" execution. Bash is calling cmd.exe to run the rails.bat file. Cmd.exe runs in ASLR mode. Used Microsoft EMET 3.0 to turn off mandatory ASLR and bottom up ASLR for both bash and cmd. Cmd.exe still runs in ASLR mode. It must be compiled that way.
Updated registry to set HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\MoveImages to 0, -1 then 1. 0 caused system problems. -1 had no effect. 1, which was supposed to be the default, helped somewhat as long as EMET changes were in place. But, it was unreliable as Address Space Occupied would eventually recur.
Decided to attempt to install a command processor that either didn’t require ASLR or allowed me to disable it using PE flags. I installed TCC/LE and attempted to get Cygwin Bash to use it to process .bat files. I tried using the path variable, the COMSPEC variable and changes to batfile and cmdfile in the registry all to no effect. Cygwin Bash must either have it hard coded or some nonobvious control.
My fix is to replace my environment. I know I will eventually migrate to some Unix, but for now I am converting to MinGW/MSYS and RubyStack. All I want to do is to get a stable RoR (Ruby on Rails) environment running for today. I don’t see that happening using Cygwin with Windows 7 Ultimate. I keep running into too many problems and this one is a very, very tough nut to crack. Thanks…
PHP has different opcode caches like APC, Zend Optimizer to cache the code and dramatically speed things up. Does Ruby have something similar?
The default Ruby 1.9.x is based on a bytecode VM, in addition you have ruby implementations based on the Java Virtual Machine (JRuby) and LLVM (Rubinius and MacRuby). These will all do just-in-time compilation and other optimizations you'd expect from a modern VM.
Default production settings in Rails is:
config.cache_classes = true
which mean that code isn't reloaded after requests, therefore it's cached in memory.
As far as MRI is concerned, experimental bytecode caching has been released with Ruby 2.3.
All you need to do to make this feature enabled is just to require
'yomikomu' rubygem and set some environmental variables introduced at
here as you can find two export commands in the example above.
It may look a bit magical why VM-level bytecode cache is enabled only
by requiring 'yomikomu' rubygem. Koichi described about this at his
ticket.
Here is a quick benchmark result of current bytecode cache implementation. I used 'bundle version' command with benchmark-ips on Ubuntu machine. Source
The post also provides some benchmarks for this newly released functionality:
$ ruby measure.rb
Comparison:
yomikomu(fs): 5.0 i/s
yomikomanai: 3.6 i/s - 1.40x slower
Other ruby implementations might be able to take advantage of the platform native optimizations - eg. JRuby benefits from the performance benefits of JVM JIT.
I mainly do ruby on rails development on my machine but from time to time I end up using other laptops for RoR development. It would be nice if there was something (maybe shell?) which basically bring all the gems installed on my machine to some other machine without leaving any footprints. It's basically a really light VM without the OS stuff.
If it matters, I'm using a mac. Ideally I would like to keep that virtual environment in my dropbox and basically when I use some other machine, I would just get it from my dropbox and start coding and not have to worry about setting up the environment.
Similar to Jacob's answer, I'd recommend using RVM, but I'll expand on it. Here's some brainstorming ideas:
RVM stores its sandbox in your home directory at ~/.rvm. All Ruby instances, plus the associated gems will be stored there. It's a simple addition to the ~/.bashrc file in your Mac to initialize RVM so it's known by the shell when you log into the account. It's also a simple rm -rf ~/.rvm from the commandline to remove the RVM sandbox from the account, followed by removing the line from the ~/.bashrc.
So, basically, by setting up RVM correctly and installing your Ruby installation on one machine, you're 90% of the way to having it available for multiple machines.
I'm pretty sure Ruby will install without any dependencies on a current Mac OS using RVM, but there's a couple libraries that can improve the experience. After installing RVM, but before installing any Rubies, run rvm notes. That will show you what else to install. You'll need the current XCode to compile a Ruby, but only on the machine you do the compiling on. Once it's installed you should be able to move a RVM controlled Ruby around to other Macs by copying the ~/.rvm directory. So, not only would you have the gems, you could have a particular version, or versions, of Ruby, plus the associated gems, so your regression tests could work too.
If you use MacVim you could install it in ~/bin and have the GUI version. I haven't tried running it from there, but it seems like it'd work. You might need to create an alias from /Applications to the one in ~/bin for double-clicking.
MacVim comes with a shell script called mvim to launch it from the command-line. I have a bunch of softlinks to mine letting me call it from the command-line in various ways: gvim, and the gvim varients like gvimdiff and gview. You could do the same by adding ~/bin to your PATH and making the links locally in that dir to MacVim's mvim.
You could build a tarball of the vim config, vim installation and RVM sandbox, copy that to another Mac, expand it, add ~/bin to your PATH and append the needed RVM initialization line in ~/.bashrc, open a new command-line, and have your editor plus Ruby sandboxes.
It's a minor variation on how my Mac and Linux boxes are set up. I haven't tried bundling everything together, but, on Macs that are the same OS version, it should work.
Consider using rvm to manage different gem configurations. If you want you can store your rvm configurations in your dropbox (rather than in ~/.rvm, where they go by default) so that your gemsets are synced across machines.