Nobody flush DB, only do hget after start and hset all the Data.
But after sometime(about 1 day) , the memory will loss and no data exists...
# Memory
used_memory:817064
used_memory_human:797.91K
used_memory_rss:52539392
used_memory_peak:33308069304
used_memory_peak_human:31.02G
used_memory_lua:36864
mem_fragmentation_ratio:64.30
mem_allocator:jemalloc-3.6.0
After I restart the server and hset all again , the used_memory recover back.
# Memory
used_memory:33291293520
used_memory_human:31.00G
used_memory_rss:33526530048
used_memory_peak:33291293520
used_memory_peak_human:31.00G
used_memory_lua:36864
mem_fragmentation_ratio:1.01
mem_allocator:jemalloc-3.6.0
But it can never last longer than 1 day...The hset process need at least 4h, and redis takes up over half of the memory so BGSAVE is useless...
What is the reason of memory loss? and how 2 backup data?
Related
I hava a Redis runing in a container .
Inside the container cgroup rss show using about 1283MB memory.
The kmem memory usage is 30.75MB.
The summary of the memory usage of all processes in the docker container is 883MB.
How can i figure out the "disappeared memory "(1296-883-30=383MB).The "disappeared memory" will growing with the time pass.Flinally the container will be oom killed .
environmet info is
redis version:4.0.1
docker version:18.09.9
k8s version:1.13
**the memory usage is 1283MB **
root#redis-m-s-rbac-0:/opt#cat /sys/fs/cgroup/memory/memory.usage_in_bytes
1346289664 >>>> 1283.921875 MB
the kmem memory usage is 30.75MB
root#redis-m-s-rbac-0:/opt#cat /sys/fs/cgroup/memory/memory.kmem.usage_in_bytes
32194560 >>> 30.703125 MB
root#redis-m-s-rbac-0:/opt#cat /sys/fs/cgroup/memory/memory.stat
cache 3358720
rss 1359073280 >>> 1296.11328125 MB
rss_huge 515899392
shmem 0
mapped_file 405504
dirty 0
writeback 0
swap 0
pgpgin 11355630
pgpgout 11148885
pgfault 25710366
pgmajfault 0
inactive_anon 0
active_anon 1359245312
inactive_file 2351104
active_file 1966080
unevictable 0
hierarchical_memory_limit 4294967296
hierarchical_memsw_limit 4294967296
total_cache 3358720
total_rss 1359073280
total_rss_huge 515899392
total_shmem 0
total_mapped_file 405504
total_dirty 0
total_writeback 0
total_swap 0
total_pgpgin 11355630
total_pgpgout 11148885
total_pgfault 25710366
total_pgmajfault 0
total_inactive_anon 0
total_active_anon 1359245312
total_inactive_file 2351104
total_active_file 1966080
total_unevictable 0
**the summary of the memory usage of all processes in the docker container is 883MB **
root#redis-m-s-rbac-0:/opt#ps aux | awk '{sum+=$6} END {print sum / 1024}'
883.609
This is happening because usage_in_bytes does not show exact value of memory and swap usage. The memory.usage_in_bytes show current memory(RSS+Cache) usage.
5.5 usage_in_bytes For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline
false sharing. usage_in_bytes is affected by the method and doesn't
show 'exact' value of memory (and swap) usage, it's a fuzz value for
efficient access. (Of course, when necessary, it's synchronized.) If
you want to know more exact memory usage, you should use
RSS+CACHE(+SWAP) value in memory.stat(see 5.2).
Reference:
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
Having a dask.DataFrame which consumes around 100GB in memory::
ddf = client.persist(ddf)
len(ddf_c.index)
# 246652596 rows
## Running some other code like groupby/aggregate etc
Now I want to filter out the data by using .loc operator, but after running the following, the RAM consumption is 165GB:
ddf_c = ddf_c.loc[ddf_c.is_in_valid_set_of_combis == True]
ddf_c = client.persist(ddf_c) # Now we have 165GB RAM consumptioon
How can I check for open/pending/waiting futures/tasks/datasets which are preventing Dask from really overwriting the ddf_c dask.DataFrame?
This is what the info page looks liks:
('loc-series-b0f23c725a607fed56584d9e41e57de8', 77) 227.41 MB
[... around 50 entries ...]
You can track dependencies in the "info" pages of the dashboard (click the info tab at the top)
In your situation I would probly not persist until the latter step.
I've tried to use Ignite to store events, but face a problem of too much RAM usage during inserting new data
I'm runing ignite node with 1GB Heap and default configuration
curs.execute("""CREATE TABLE trololo (id LONG PRIMARY KEY, user_id LONG, event_type INT, timestamp TIMESTAMP) WITH "template=replicated" """);
n = 10000
for i in range(200):
values = []
for j in range(n):
id_ = i * n + j
event_type = random.randint(1, 5)
user_id = random.randint(1000, 5000)
timestamp = datetime.datetime.utcnow() - timedelta(hours=random.randint(1, 100))
values.append("({id}, {user_id}, {event_type}, '{timestamp}')".format(
id=id_, user_id=user_id, event_type=event_type, uid=uid, timestamp=timestamp.strftime('%Y-%m-%dT%H:%M:%S-00:00')
))
query = "INSERT INTO trololo (id, user_id, event_type, TIMESTAMP) VALUES %s;" % ",".join(values)
curs.execute(query)
But after loading about 10^6 events, I got 100% CPU usage because all heap are taken and GC trying to clean some space (unsuccessfully)
Then I stop for about 10 minutes and after that GC succesfully clean some space and I could continue loading new data
Then again heap fully loaded and all over again
It's really strange behaviour and I couldn't find a way how I could load 10^7 events without those problems
aproximately event should take:
8 + 8 + 4 + 10(timestamp size?) is about 30 bytes
30 bytes x3 (overhead) so it should be less than 100bytes per record
So 10^7 * 10^2 = 10^9 bytes = 1Gb
So it seems that 10^7 events should fit into 1Gb RAM, isn't it?
Actually, since version 2.0, Ignite stores all in offheap with default settings.
The main problem here is that you generate a very big query string with 10000 inserts, that should be parsed and, of course, will be stored in heap. After decreasing this size for each query, you will get better results here.
But also, as you can see in doc for capacity planning, Ignite adds around 200 bytes overhead for each entry. Additionally, add around 200-300MB per node for internal memory and reasonable amount of memory for JVM and GC to operate efficiently
If you really want to use only 1gb heap you can try to tune GC, but I would recommend increasing heap size.
In fluentd, regarding retry_limit, disable_retry_limit http://docs.fluentd.org/v0.12/articles/output-plugin-overview:
If the limit is reached, buffered data is discarded and the retry interval is reset to its initial value (retry_wait).
In my setup I have the following configuration for output:
buffer_queue_limit 200
buffer_chunk_limit 1m
flush_interval 3s
buffer_queue_full_action drop_oldest_chunk
max_retry_wait 1h
disable_retry_limit true
So we will keep retrying to output from buffer, with a max_retry_wait of 1 hour, untill the buffer queue is full, in which case it will drop the oldest chunk and move onto the next one.
With the disable_retry_limit set to true, this means we drop the oldest chunk only when the buffer queue is full, buffer_queue_full_action drop_oldest_chunk.
My question is, when this buffer queue drops the oldest chunk, is the retry_wait(default 1s, incrementing with each try) reset to it's initial value for the next chunk in the queue due to be outputted (giving same behavior as when retry_limit is reached)
Tested on local machine, fluent-d does not reset the retry_wait to its initial value when a chunk is dropped.
The redis server version I use is 2.8.9 from MSOpenTech github. Can anyone shed light on why redis "info" command indicates that used memory is 21 GB even though the RDB file that's saved on disk is < than 4 GB? I did successfully run a "save" command before noting down the size of the RDB file. The qfork heapfile is 30 Gb as it's been configured in redis.windows.conf.
Configuration :
maxheap 30gb
max-memory 20 Gb
appendonly no
save 18000 1
The server has 192 GB of physical RAM, but unfortunately only has about 60 GB of free disk space and I had to set max-heap and max-memory to 30 Gb and 20 Gb respectively so that I have additional space to persist the data on disk.
I'm using redis as a cache and the save interval is large as seeding the data takes a long time and I don't want constant writing to file. Once seeding is done, the DB is updated with newer data once a day.
My questions are :
How is the saved RDB file so small? Is it solely due to compression (rdbcompression yes)? If yes, can the same compression mechanism be used to store data in memory too? I make use of lists extensively.
Before I ran the "save" command, the working set and private bytes in process-explorer was very small. Is there a way I can breakdown memory usage by datastructure? For example : List uses x amount, Hash uses y amount etc?
Is there any way I can store the AOF file ( I turned off AOF and use RDB because the AOF files were filling up disk space fast ) in a network path ( shared drive or NAS )? I tried setting the dir config to \someip\some folder but the service failed to start with the message "Cant CHDIR to location"
I'm unable to post images, but this is what process-explorer has to say about the redis-server instance :
Virtual Memory:
Private Bytes : 72,920 K
Peak Private Bytes : 31,546,092 K
Virtual Size : 31,558,356 K
Page faults : 12,479,550
Physical Memory:
Working Set : 26,871,240 K
WS Private : 63,260 K
WS Shareable : 26,807,980 K
WS Shared : 3,580 K
Peak Working Set : 27,011,488 K
The latest saved dump.rdb is 3.81 GB and the heap file is 30 GB.
# Server
redis_version:2.8.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:1fe181ad2447fe38
redis_mode:standalone
os:Windows
arch_bits:64
multiplexing_api:winsock_IOCP
gcc_version:0.0.0
process_id:12772
run_id:553f2b4665edd206e632b7040aa76c0b76083f4d
tcp_port:6379
uptime_in_seconds:24087
uptime_in_days:0
hz:50
lru_clock:14825512
config_file:D:\RedisService/redis.windows.conf
# Clients
connected_clients:2
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:21484921736
used_memory_human:20.01G
used_memory_rss:21484870536
used_memory_peak:21487283360
used_memory_peak_human:20.01G
used_memory_lua:3156992
mem_fragmentation_ratio:1.00
mem_allocator:dlmalloc-2.8
# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1407328559
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:1407328560
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:9486
total_commands_processed:241141370
instantaneous_ops_per_sec:0
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:30143
keyspace_misses:81
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:1341134