Flutter app "documents and data" disk usage is too high on iOS - dart

I have written a very simple app in Flutter, for both iOS, Android and the web.
Lately, I realized that if I browse the "storage" settings page on my iPhone, my prod app, as it is distributed on the App Store, weighs about 500 MB. Most of it comes from the "documents and data" part, the app itself isn't huge:
Storage page
This is unexpected as the only thing I am storing on purpose is a small sqlite database, which in my case is about 30 KB.
Following this guide from the Apple support page, I was able to download my app's container, which is indeed about 500 MB. It turns out most of the weight comes from inside of a tmp folder, as shown by the output of du -sh AppData/tmp/*:
0B AppData/tmp/count0fSKCX
0B AppData/tmp/count1N2yFY
32K AppData/tmp/count1tKqnr
0B AppData/tmp/count2BxlSk
24K AppData/tmp/count2VKOVX
0B AppData/tmp/count2tnzwn
[...]
0B AppData/tmp/count8kl1hK
53M AppData/tmp/count8kqOke
0B AppData/tmp/count8ssdC7
[...]
0B AppData/tmp/countZHwkA9
26M AppData/tmp/countZHx1v8
53M AppData/tmp/countZKP9JU
0B AppData/tmp/counta5fYmx
[...]
If I take a look at what's inside of one of those huge 50+ MB folders, here is what takes so much space:
du -sh AppData/tmp/countZKP9JU/count/*
26M AppData/tmp/countZKP9JU/count/main.dart.dill
20K AppData/tmp/countZKP9JU/count/main.dart.incremental.dill
26M AppData/tmp/countZKP9JU/count/main.dart.swap.dill
I failed to find useful documentation about those files, as I am not sure what to look for: is the problem in my Dart config, in my Flutter config, in my App config, ...? Can you guys please enlighten me?
Edit: Here is some version info that might be useful
Flutter 2.5.1 • channel stable • https://github.com/flutter/flutter.git
Framework • revision ffb2ecea52 (5 weeks ago) • 2021-09-17 15:26:33 -0400
Engine • revision b3af521a05
Tools • Dart 2.14.2
iOS version: 15.0.2

In case someone runs into the same issue, I ended up writing my own cleanup logic on startup, calling getApplicationDocumentsDirectory().parent.list() and then deleting the tmp child folder if it exists.
I don't know if this is necessary at all in the end, as this tmp folder growing infinitely might only happen because of the development builds I keep installing on my phone. But this cleanup step probably won't hurt anyway.
Edit: Here is the code. Feel free to improve it, I think it probably logs an exception on other platforms (but it doesn't crash):
import 'package:path_provider/path_provider.dart';
[...]
#override
void initState() {
_cleanUpTemporaryDirectory();
super.initState();
}
[...]
_cleanUpTemporaryDirectory() async {
final documentsDirectory = await getApplicationDocumentsDirectory();
documentsDirectory.parent.list().forEach((child) async {
if (child is Directory && child.path.endsWith('/tmp')) {
print('Deleting temp folder at ${child.path}...');
try {
await child.delete(recursive: true);
print('Temp folder was deleted with success');
} catch (error) {
print('Temp folder could not be deleted: $error');
}
}
});
}

Related

How can I fix a "bad message" in NixOS?

This is what happens when I run nix-collect-garbage:
jon#jon-laptop ~> nix-collect-garbage -d
removing old generations of profile /nix/var/nix/profiles/per-user/jon/profile
removing old generations of profile /nix/var/nix/profiles/per-user/jon/home-manager
removing old generations of profile /nix/var/nix/profiles/per-user/jon/channels
finding garbage collector roots...
0 store paths deleted, 0.00 MiB freed
error: --- Error ---------------------------------------------------------------- nix-collect-garbage
reading directory '/nix/store/trash/apni99c9rdaii9xvsmf1x65dvwgsaxj0-sqlite-3.33.0-bin/share/man': Bad message
How can I fix this? I can't just delete the offending directory, apparently, since it tells me it's a read-only filesystem.

How to debug headless pdf printing problems in chrome?

Note: this is not (directly) a question about how to print PDF in chrome, instead this is a question about how to get more information when printing fails.
In short: I cannot solve a printing PDF problem, which occurs only for certain (presumably large) pages and could use some assistance in debugging the actual issue.
Background: I am using the chromedriver (v83) and chromium-browser (v83) to print PDF files from webpages by utilizing python selenium. I am building a docker image to contain the required dependencies for this. I have tried to use Debian (buster and stretch) as well as Alpine base images, but all eventually result in the same error, when trying to print some pages. The odd thing is that for other (smaller) pages the printing works, but when many assets and pages are to be printed, the printing fails. I might add that this docker images is eventually being run inside of a Kubernetes cluster, where I assigned up to 4GB of RAM.
What code am I running?
This project was written for python3, so here are some relevant code fragments. Please note that I removed all error handling and waiting for the page loads to complete here.
from selenium import webdriver
appState = {
"recentDestinations": [
{
"id": "Save as PDF",
"origin": "local"
}
],
"selectedDestinationId": "Save as PDF",
"version": 2
}
def get_chrome_options(headless: bool, enable_logging: bool) -> Options:
chrome_options = webdriver.ChromeOptions()
profile = {'printing.print_preview_sticky_settings.appState': json.dumps(appState)}
chrome_options.add_experimental_option('prefs', profile)
if headless:
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--window-size=1920,1080')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--disable-web-security')
chrome_options.add_argument('-–allow-file-access-from-files')
chrome_options.add_argument('--run-all-compositor-stages-before-draw')
chrome_options.add_argument('--kiosk-printing')
if enable_logging:
chrome_options.add_argument('--enable-logging')
return chrome_options
def print_the_page(url):
driver = webdriver.Chrome(chrome_options=get_chrome_options(headless, enable_logging))
driver.execute(driver_command=Command.GET, params={'url': url})
command_url = f"{driver.command_executor._url}/session/{driver.session_id}/chromium/send_command_and_get_result"
response = driver.command_executor._request('POST', command_url, json.dumps({'cmd': 'Page.printToPDF', 'params': {}}))
Then what happens?
For some pages this fails - meaning - there is this message in the response:
{'status': 500, 'value': '{"value":{"error":"unknown error","message":"unknown error: unhandled inspector error: {\\"code\\":-32000,\\"message\\":\\"Printing failed\\"}\\n (Session info: headless chrome=83.0.4103.116)","stacktrace":""}}'}
[UPDATE]
I have managed to produce some more error output when using the --print-to-pdf option directly, which seems to hint at an "out-of-memory" issue here:
[0923/135406.102857:WARNING:discardable_shared_memory_manager.cc(194)] Less than 64MB of free space in temporary directory for shared memory files: 23
[0923/135406.110108:WARNING:dns_config_service_posix.cc(341)] Failed to read DnsConfig.
[0923/135406.180892:WARNING:dns_config_service_posix.cc(341)] Failed to read DnsConfig.
[0923/135406.613221:FATAL:memory.cc(38)] Out of memory. size=796176
Received signal 6
r8: 00007fa6f39dadc4 r9: 0000000000000000 r10: 0000000000000008 r11: 0000000000000246
r12: 0000557efd1b0660 r13: 0000000000000000 r14: 00007fa6f39db240 r15: 0000000000000043
di: 0000000000000002 si: 00007fa6f39dac90 bp: 00007fa6f39dac90 bx: 0000000000000000
dx: 0000000000000000 ax: 0000000000000000 cx: 00007fa6fd347a71 sp: 00007fa6f39dac88
ip: 00007fa6fd347a71 efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000
trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.
[0923/135406.626313:ERROR:headless_shell.cc(399)] Abnormal renderer termination.
I will note here that I have been running this docker container locally on my machine (which has more than enough RAM) as well as on a Kubernetes cluster where this image has requested 4GB RAM. I also monitored the RAM usage and it didn't seem to be an issue - although - that could be illusive if the RAM usage is so radically high that chrome just fails and you never really see that in the overall RAM usage.
[UPDATE 2]
I have tried to use the --print-to-pdf option again, but I am seeing issues with that as well. The resources are loading, but the printing still fails.
│ [0923/144355.169080:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
...
│ [0923/141758.393923:WARNING:dns_config_service_posix.cc(341)] Failed to read DnsConfig. │
│ [0923/141758.401925:ERROR:zygote_host_impl_linux.cc(262)] Failed to adjust OOM score of renderer with pid 32: Permission denied (13) │
│ [0923/141758.413475:ERROR:zygote_host_impl_linux.cc(262)] Failed to adjust OOM score of renderer with pid 36: Permission denied (13)
... loading all the resources ...
│ [0923/141824.611661:ERROR:print_render_frame_helper.cc(1889)] Printing failed. │
│ [0923/141824.612439:ERROR:headless_shell.cc(562)] Print to PDF failed
What's the question(s)?
How can I get more information about why the "Printing failed" - unfortunately the "unknown error: unhandled inspector error" hasn't given me any ideas about how to proceed.
Are there potentially any additional flags to get more debug output from chrome or is there a log somewhere that I should be able to find?
What else have I tried?
I have initially been running this under Debian buster with the latest google-chrome and chromium binaries (v85). I have switched to an Alpine base image and chromium - hoping that this might change something, but it didn't.
I have experimented with setting up Xvfb ${DISPLAY} -screen ${SCREEN} ${RESOLUTION} & in Docker, but it didn't seem to have any effect either.
I have tried to switch to using the direct cli google-chrome --print-to-pdf= option, but since it's a page that requires to pass through login authentication, I could only get the login page to print and it also seems to have some not so nice formatting issues.
I have been running this on my machine, outside of Docker, and was able to print as expected, but as soon as I put the same code inside a Docker container, it fails.
Unfortunately, I cannot share the page where this fails with you.
The relevant warning from your logs seems to be this:
[0923/135406.102857:WARNING:discardable_shared_memory_manager.cc(194)] Less than 64MB of free space in temporary directory for shared memory files: 23
The problem appears to stem from Docker's mounted /dev/shm being too small for Chromium to do things like you're trying to do.
I found a closed bug report against chromium referencing this issue in certain limited environments such as AWS Lambda and Docker, it was fixed in chromium v65 behind a command line flag --disable-dev-shm-usage.
The last few comments reference another bug report (now closed) about this issue in chromium v83 where the command line flag was not working properly. It has been fixed in version 84 - per comment 28:
You can find the fix in current stable release of Chrome (version 84.0.4147.89 and above).
You've indicated you're using chromium v83, so you'll need to update at least version 84.0.4147.89, then use the command line flag --disable-dev-shm-usage.

"wasm decoding failResult" when app is packaged with electron-forge

I am trying to package an electron app (that uses wasm) with electron-forge.
At the moment (while I fight packaging!) it's little more than a hello-world, but it compiles a simple rust lib to wasm, then I copy the .js and .wasm files into ./src/ and package as an electron app.
Running it using "electron-forge start" works perfectly, but when I package it with "electon-forge make" (to a DMG on Mac OS in this case) and then install and run it, it seems to be trying to load the wrong .wasm file (or it's been modified...) as I get these errors:
There is only one .wasm file bundled, and I have checked the .js that it has the reference to the correct file.
If I go into the .app package in /Applications and navigate to the packaged source files all is correct, and the same as my source folders.
Checking the magic number of the .wasm file also seems fine..
hexdump ui-72b0082cbbe1a2b5.wasm |head -n 1
0000000 00 61 73 6d 01 00 00 00 01 86 01 13 60 02 7f 7f
So, I don't know what file it's trying to load as .wasm and why the magic number is not correct.
Any ideas?
Not sure if you still have this problem, but this was stumping me for a while, so I figured I'd leave the answer for the next person. Short answer, you need to add a small bit to your json config file. Example:
Default config:
...
"config": {
"forge": {
"packagerConfig": { },
...
Fixed config:
...
"config": {
"forge": {
"packagerConfig": { "asar.unpack" : "yourFileHere.js" },
...
Best I can tell, the problem is coming from electron-packager and how it compresses everything into that asar file. It seems to try and serve the compressed file, so the MIME type is wrong. But even when I changed the web assembly loading code to use WebAssembly.instantiate instead of WebAssembly.instantiateStreaming (that avoids the MIME check), I was still left with those magic number issues...that was the clue that compression was the issue.
You can do directories also, info here about other packager options
https://github.com/electron/electron-packager/blob/master/docs/api.md#asar

Grunt tasks are slow in yeoman application

I have an angular project build with yeoman, talking to a rails api backend.
Everything works fine, except that grunt tasks are very slow.
When I run grunt server --verbose:
Execution Time (2014-01-15 13:37:55 UTC)
loading tasks 14.3s ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 26%
server 1ms 0%
preprocess:multifile 11ms 0%
clean:server 13ms 0%
concurrent:server 34.3s ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 63%
autoprefixer 1ms 0%
autoprefixer:dist 369ms ▇ 1%
connect:livereload 17ms 0%
watch 5.8s ▇▇▇▇▇▇▇▇▇ 11%
Total 54.8s
Some of my Gruntfile:
'use strict';
module.exports = function (grunt) {
require('time-grunt')(grunt);
require('load-grunt-tasks')(grunt);
require('time-grunt')(grunt);
grunt.initConfig({
...
});
grunt.loadNpmTasks('grunt-preprocess');
grunt.registerTask('server', function (target) {
if (target === 'dist') {
return grunt.task.run(['build', 'connect:dist:keepalive']);
}
grunt.task.run([
'preprocess:multifile',
'clean:server',
'concurrent:server',
'autoprefixer',
'connect:livereload',
'watch'
]);
});
grunt.registerTask('test', [
'clean:server',
'concurrent:test',
'autoprefixer',
'connect:test'
//'karma'
]);
grunt.registerTask('build', [
'preprocess:multifile',
'clean:dist',
'useminPrepare',
'concurrent:dist',
'autoprefixer',
'concat',
'copy:dist',
'cdnify',
'ngmin',
'cssmin',
'uglify',
'rev',
'usemin'
]);
grunt.registerTask('default', [
'jshint',
'test',
'build'
]);
};
Size of project:
vagrant#vm ~code/myapp/app/scripts
$> find -name "*.js" | xargs cat | wc -l
10209
I am running on MacOS 10.8 with i7 processor, 16GB ram, SSD... It is normal that is takes so long ? What makes the grunt task (and especially "loading tasks") so slow ?
Note: I am ssh'd inside a vagrant machine and running the grunt commands from there. If I run the grunt command on my native system, it's much faster (loading tasks takes 1.6s instead of 14.3).
So the shared filesystem might be an issue. But why...
I had exactly same problem with Vagrant and Yeomans angular-generator. After running grunt serve it took almost 30 seconds to compile sass, restart server etc.
I already used NFS but it was still slow. Then I tried jit-grunt, just-in-time grunt loader. I replaced load-grunt-tasks with jit-grunt and everything is now a lot faster.
Here's a good article about JIT-Grunt:
https://medium.com/written-in-code/ced193c2900b
I am using grunt inside a Vagrant virtualbox. (ubuntu 12.04). My native files are on my host machine (OSx). Because the grunt tasks are io-intensive, and they are run through file sharing, this make them quite slow.
This can be improved by adding nfs to Vagrant (http://docs.vagrantup.com/v2/synced-folders/nfs.html). This will make Vagrant share files with nfs instead of the default Vagrant file sharing. It will be a bit faster, but not much.
For comparison, on my machine:
For running the subtask loading grunt tasks
natively: 1.2s
with nfs: 4s
vagrant file sharing: 16s
If only a specific task is taking a lot of time, this specific task might be the problem. To troubleshoot, use time-grunt: https://npmjs.org/package/time-grunt.
I've had issues as well, and have found:
nospawn: true
To be the fastest option. I went from ~20s to ~1s to concat, minify, and uglify JS.
I had the same problem with Yeoman's ngbp generator and Vagrant. Even with nfs, a simple change on a template took about 30s to be seen in the browser.
Using jit-grunt caused time to be reduced to 10s. After using spawn:false, even though there was no reduction at the first load, changes took less than 1s (0.086s) to propagate to the browser! (Yes!)
The changes I've made to Gruntfile.js:
I commented all the grunt.loadNpmTasks but grunt.loadNpmTasks('grunt-contrib-watch') [That's because of a task rename ngbp does later on];
I added require('jit-grunt')(grunt); after grunt.loadNpmTasks('grunt-contrib-watch');
I added spawn: false to delta: { options: {livereload: true, spawn: false} ...}.

CouchDB 1.3.1 on Centos 6.4

I compiled CouchDB and installed. It seems to work great except when I use views on the database, then it just spins the wheel and nothing happens and the cpu load spikes to 100% and slowly it eats away all memory and starts to swap a lot which in return increases the cpu load.
I have tried both with the js-1.70-12 that comes with centos 6.4, as well as build and install my own js-1.85-1. All erlang packages are installed from epel :
erlang-crypto-R14B-04.2.el6.x86_64
erlang-syntax_tools-R14B-04.2.el6.x86_64
erlang-mnesia-R14B-04.2.el6.x86_64
erlang-ssl-R14B-04.2.el6.x86_64
erlang-cosProperty-R14B-04.2.el6.x86_64
erlang-asn1-R14B-04.2.el6.x86_64
erlang-cosEventDomain-R14B-04.2.el6.x86_64
erlang-eunit-R14B-04.2.el6.x86_64
erlang-erl_docgen-R14B-04.2.el6.x86_64
erlang-toolbar-R14B-04.2.el6.x86_64
erlang-debugger-R14B-04.2.el6.x86_64
erlang-tools-R14B-04.2.el6.x86_64
erlang-typer-R14B-04.2.el6.x86_64
erlang-megaco-R14B-04.2.el6.x86_64
erlang-oauth-1.1.1-1.el6.x86_64
erlang-stdlib-R14B-04.2.el6.x86_64
erlang-hipe-R14B-04.2.el6.x86_64
erlang-kernel-R14B-04.2.el6.x86_64
erlang-runtime_tools-R14B-04.2.el6.x86_64
erlang-snmp-R14B-04.2.el6.x86_64
erlang-public_key-R14B-04.2.el6.x86_64
erlang-inets-R14B-04.2.el6.x86_64
erlang-ibrowse-2.2.0-4.el6.x86_64
erlang-cosEvent-R14B-04.2.el6.x86_64
erlang-cosNotification-R14B-04.2.el6.x86_64
erlang-edoc-R14B-04.2.el6.x86_64
erlang-otp_mibs-R14B-04.2.el6.x86_64
erlang-cosFileTransfer-R14B-04.2.el6.x86_64
erlang-cosTransactions-R14B-04.2.el6.x86_64
erlang-inviso-R14B-04.2.el6.x86_64
erlang-jinterface-R14B-04.2.el6.x86_64
erlang-erl_interface-R14B-04.2.el6.x86_64
erlang-diameter-R14B-04.2.el6.x86_64
erlang-gs-R14B-04.2.el6.x86_64
erlang-tv-R14B-04.2.el6.x86_64
erlang-appmon-R14B-04.2.el6.x86_64
erlang-odbc-R14B-04.2.el6.x86_64
erlang-wx-R14B-04.2.el6.x86_64
erlang-et-R14B-04.2.el6.x86_64
erlang-observer-R14B-04.2.el6.x86_64
erlang-sasl-R14B-04.2.el6.x86_64
erlang-dialyzer-R14B-04.2.el6.x86_64
erlang-common_test-R14B-04.2.el6.x86_64
erlang-os_mon-R14B-04.2.el6.x86_64
erlang-examples-R14B-04.2.el6.x86_64
erlang-compiler-R14B-04.2.el6.x86_64
erlang-erts-R14B-04.2.el6.x86_64
erlang-xmerl-R14B-04.2.el6.x86_64
erlang-orber-R14B-04.2.el6.x86_64
erlang-cosTime-R14B-04.2.el6.x86_64
erlang-ssh-R14B-04.2.el6.x86_64
erlang-docbuilder-R14B-04.2.el6.x86_64
erlang-percept-R14B-04.2.el6.x86_64
erlang-parsetools-R14B-04.2.el6.x86_64
erlang-ic-R14B-04.2.el6.x86_64
erlang-pman-R14B-04.2.el6.x86_64
erlang-webtool-R14B-04.2.el6.x86_64
erlang-test_server-R14B-04.2.el6.x86_64
erlang-reltool-R14B-04.2.el6.x86_64
erlang-R14B-04.2.el6.x86_64
erlang-mochiweb-1.4.1-5.el6.x86_64
Every thing configures and makes and installs as expected. You can dump data into the database, you can create documents and all that. But I can not run any view, temporary or not.
The only error I see in the logs is like this one, and it is a lot of these errors :
[Sun, 18 Aug 2013 23:10:38 GMT] [error] [<0.124.0>] {error_report,<0.30.0>,
{<0.124.0>,crash_report,
[[{initial_call,
{mochiweb_socket_server,init,['Argument__1']}},
{pid,<0.124.0>},
{registered_name,[]},
{error_info,
{exit,eaddrinuse,
[{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,
[couch_secondary_services,couch_server_sup,
<0.31.0>]},
{messages,[]},
{links,[<0.93.0>]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,987},
{stack_size,24},
{reductions,459}],
[]]}}
But I have no idea what they mean.
Do I need to compile and install erlang as well ? All the above packages or just erlang ?
Your compilation and installation looks fine. At least your error (note: eaddrinuse in traceback) is about that there is some process that listens same address and port as your CouchDB try to. Check other listening processes with netstat -anp command or change CouchDB's listen port to different.

Resources