lot of temp magick files created in temporary folder - imagemagick

I am using the imagick library to resizing and cropping the images in a http handler. Which doesn't write anything in /tmp folder. But as i can a lot of these files are being created in that folder and it's size is growing on a daily basis and currently it's consuming around 90% of partition size. Moreover, i am not able to read their content also. So, why these files are being created and would they be delete after a while or do i need to manually delete them.
-rw------- 1 ubuntu ubuntu 16M Feb 22 09:46 magick-d6ascKJV
-rw------- 1 ubuntu ubuntu 16M Feb 22 09:46 magick-46ccZIfq
-rw------- 1 ubuntu ubuntu 1.8M Feb 22 09:47 magick-vUET7vyh
-rw------- 1 ubuntu ubuntu 1.8M Feb 22 09:47 magick-OLkGWTX8
-rw------- 1 ubuntu ubuntu 15M Feb 22 09:48 magick-LNMV7YvE
-rw------- 1 ubuntu ubuntu 16M Feb 22 09:49 magick-0LMYt6Kc
-rw------- 1 ubuntu ubuntu 16M Feb 22 09:50 magick-ceNxX5CY
-rw------- 1 ubuntu ubuntu 16M Feb 22 09:50 magick-nQ1M3y6I
Edit :
I am not using the following two lines in my http Handler. Reason being i couldn't find any explanation to do so. Moreover, the go http handler works fine. So, what is the purpose of these statements?
imagick.Initialize()
defer imagick.Terminate()
I am assuming there would be some reason to include them in the code. So, in go http handler. Where it should be included inside func main() or inside serveHTTP ?
func main() {
myMux := http.NewServeMux()
myMux.HandleFunc("/", serveHTTP)
if err := http.ListenAndServe(":8085", myMux); err != nil {
logFatal("Error when starting or running http server: %v", err)
}
}
func serveHTTP(w http.ResponseWriter, r *http.Request) {
}

Temporary / intermediate files should be automatically cleaned-up by the calling ImageMagick thread. The pattern-behavior you are describing hints at a bug in the application using ImageMagick.
I would suggest...
Check error logs of application.
Evaluate error reporting of calling code.
Ensure proper imagick.Initialize & imagick.Terminate routines are called
And if nothing else works, use the environment variable MAGICK_TMPDIR to control where the imagemagick artifacts are written to.
Update
The imagick.Initialize wraps the underlying MagickCoreGenesis, and imagick.Terminate the MagickCoreTerminus routine. They are important for managing the environment in which ImageMagick operates. They should be called on the worker thread that will handle any ImageMagick tasks, so in your case, serveHTTP method. BUT it would not be optimal to perform such routines with every HTTP request, and an additional condition should be evaluated -- if possible.
func serveHTTP(w http.ResponseWriter, r *http.Request) {
// if request will do image work
imagick.Initialize()
defer imagick.Terminate()
// ... image methods ...
// end if
}

You can set a cron which runs in a day or after an hour cleaning up your temp folder for all temporary magick files. you can use this on line command to delete all magick files as well
sudo find /tmp/ -name "magick-*" -type f -delete

Actually it should be included in the func main() not in the
func serveHTTP(). It is meant to be called once for a long running application and it solved my problem.
func main() {
imagick.Initialize()
defer imagick.Terminate()
myMux := http.NewServeMux()
myMux.HandleFunc("/", serveHTTP)
if err := http.ListenAndServe(":8085", myMux); err != nil {
logFatal("Error when starting or running http server: %v", err)
}
}
func serveHTTP(w http.ResponseWriter, r *http.Request) {
}

Related

uwsgi upgrade to python3.7 to fix ImportError: No module named 'encodings'

Honestly, I have no idea what I am doing, so be gentle with me. I am trying to use uwsgi to run my django application on a aws ubuntu instance. I have a virtual environment with python3.7 running, but when I try to run uwsgi. I get this in the logs:
*** Starting uWSGI 2.0.14 (64bit) on [Sun Jan 5 14:51:32 2020] ***
compiled with version: 5.4.0 20160609 on 20 October 2016 05:56:34
os: Linux-4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018
nodename: ip-172-31-41-139
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
chdir() to /home/ubuntu/web/graff
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 3804
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /home/ubuntu/web/graffuwsgi.sock fd 3
Python version: 3.5.2 (default, Oct 8 2019, 13:06:37) [GCC 5.4.0 20160609]
Set PythonHome to /home/ubuntu/.virtualenvs/graff
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'
Here is my uwsgi.conf
# file: /etc/init/graffuwsgi.conf
description "uWSGI server for graff"
start on runlevel [2345]
stop on runlevel [!2345]
respawn
exec /usr/local/bin/uwsgi --home /home/ubuntu/web/graff/ --socket /home/ubuntu/web/graffuwsgi.sock --chmod-socket=666 --module=graff.wsgi --pythonpath /home/ubuntu/web -H /home/ubuntu/.virtualenvs/graff --logto /home/ubuntu/web/logs/graffuwsgi.log --chdir=/home/ubuntu/web/graff --chmod-socket=666
It seems like python3.5 just doesn't work anymore. I feel like I've had to replace python3.5 with 3.7 in several places lately to fix various bugs, and I have it in my head that if I can get uwsgi to run python3.7 instead of 3.5 then that will solve this error too. Anyway, any help is much appreciated.
Looks like your uwsgi is compiled with different python version, make sure you compile with python 3.5
PYTHON=python3.5 uwsgi --build-plugin "/usr/src/uwsgi/plugins/python python35"
mv python35_plugin.so /usr/lib/uwsgi/plugins/python35_plugin.so
chmod 644 /usr/lib/uwsgi/plugins/python35_plugin.so
you can follow this guide:
https://www.paulox.net/2017/04/04/how-to-use-uwsgi-with-python3-6-in-ubuntu/
The source of error is PythonHome (pyhome, venv, home) setting.
See at the official Python docs on Environment variables
Is your path /home/ubuntu/.virtualenvs/graff suited for Python requirements?
In short:
Use pythonpath (pp) and let the system to find modules.
You can set several repeated options in your uwsgi config to custom modules search, eg:
pythonpath = /opt/web2py/
pythonpath = /opt/anaconda/lib/python3.9/site-packages/
So, I had the same error. And it was fixed by editing uwsgi vassal config.
I thing the core of problem that virtual envs are created with symlinks or have some inner relative paths, so an isolated process inside uwsgi could not find modules.

How does /dev work in Docker?

I had some issues with /dev/random and I replaced it with /dev/urandom on some of my servers:
lrwxrwxrwx 1 root root 12 Jul 22 21:04 /dev/random -> /dev/urandom
I since have replaced some of my infrastructure with Docker. I thought, it would be sufficient to replace /dev/random on my host machine. But when starting the service, I quickly noticed that some RNG operations blocked because it used the original implementation of /dev/random.
I wrote a little test program to proof this behavior:
package main
import (
"fmt"
"os"
)
func main() {
f, err := os.Open("/dev/random")
if err != nil {
panic(err)
}
chunk := make([]byte, 1024)
index := 0
for {
index = index + 1
_, err := f.Read(chunk)
if err != nil {
panic(err)
}
fmt.Println("iteration ", index)
}
}
Executing this program on the host machine (which has the symlink) works as expected - it will not block and run until I shut it down.
When running this in a container, it will block after the first iteration (at least on my machine).
I can obviously fix this problem by mounting my random file into the container:
docker run -it --rm -v /dev/random:/dev/random bla
But this is not the point of this question. I want to know, the following:
How does Docker set up the devices listed in /dev
Why is it not just using (some) of the device files of the host machine.
Docker never uses any of the host system's filesystem unless you explicitly instruct it to. The device files in /dev are whatever (if anything) is baked into the image.
Also note that the conventional Unix device model is that a device "file" is either a character- or block-special device, and a pair of major and minor device numbers, and any actual handling is done in the kernel. Relatedly, the host's kernel is shared across all Docker containers. So if you fix your host's broken implementation of /dev/random then your containers will inherit this fix too; but merely changing what's in the host's /dev will have no effect.

How messages from printf are routed to the terminal?

Lets say I have opened two tabs in the konsole (Tab1 and Tab2).
When I run tty in both of them I have:
Tab1:
~$ tty
/dev/pts/23
Tab2:
~$ tty
/dev/pts/24
If I run a simple program hello.c with a printf("Hello") in Tab1, how the system goes from writing to the stdout (file id 1) to writing to /dev/pts/23, being read by the konsole and then appearing in Tab1?
How the system know it has to give the "Hello" string to /dev/pts/23 and not to /dev/pts/24? And how it does that?
Is there a parameter given by the bash to the program so it knows which psudoterminal to send the "Hello"? Or the program sends the string back to the bash (how?) who knows to which pseudoterminal to send the data?
Thank you for your help
If you look at your process open files, you can see that the STDOUT,STDERR, etc points to the specific psuedo terminal that you already figured out using tty in your question
root#hello:~# ls -l /proc/self/fd
total 0
lrwx------ 1 root root 64 May 21 02:18 0 -> /dev/pts/3
lrwx------ 1 root root 64 May 21 02:18 1 -> /dev/pts/3
lrwx------ 1 root root 64 May 21 02:18 2 -> /dev/pts/3
As you might know, a process is created by a fork system call that actually duplicates the open file descriptors from the parent. so basically, your process gets the file descriptors from its parent.
How did the parent hot these associated with him ? well, konsole already dealt with that.

What is the tar-data.json.gz in /var/lib/docker/graph/<image GUID>

I use the Docker 1.8.3. The /var/lib/docker/repositories-aufs stores the local images information matching the output of # docker images. The /var/lib/docker/graph/<image GUID> maintain metadata and so on for each images.
The json holds metadata about the image.
The layersize indicates the size of the layer.
And what is the tar-data.json.gz and v1Compatibility?
# ll /var/lib/docker/graph/ca0ef69
drwx------ 2 root root 4096 10月 29 12:08 ./
drwx------ 150 root root 20480 11月 1 12:29 ../
-rw------- 1 root root 1384 10月 29 12:06 json
-rw------- 1 root root 1 10月 29 12:06 layersize
-rw------- 1 root root 82 10月 29 12:06 tar-data.json.gz
-rw------- 1 root root 1384 10月 29 12:08 v1Compatibility
tar-data.json.gz stores the image layer:
This seems to have been introduced in docker 1.8 by PR 14067.
This pull request introduces a library (vbatts/tar-split) that is for inline disassembly of TAR archives.
The disassembly does not do any extraction, but preserves the raw bytes of headers and padding from the archive, and defers the extraction. This way validation and extraction continue to be a lockstep process.
For an docker pull, docker load or docker commit after this feature, there will be a new state file stored for the image layer (e.g. /var/lib/docker/graph/<ID>/tar-data.json.gz).
For existing images that do not have this new state file, the tar archive produced by docker save or docker push will fallback to the traditional graphdriver.Diff as they have been.
The benefit of this feature is that rather than hoping the graph.TarLayer produced is deterministic, the tar archive will be reassembled from the raw bytes of the original archive.
Presently the issue exists that an image pulled from a repo like the Docker hub, and then pushed to a local registry may likely have a new digests.
v1compatibility was introduced in docker 1.3 (commit 15d5c7f), but is used only with docker 1.8 (commit 745820f) in manifest.go
// History stores unstructured v1 compatibility information
type History struct {
// V1Compatibility is the raw v1 compatibility information
V1Compatibility string `json:"v1Compatibility"`
}
And really used in docker 1.9 (commit 504e67b) in graph/graph.go, where the v1Compatibility JSON data associated
with the image in the manifest is stored to the disk.
Then graph/pull_v2.go, when attempting a tag reuse, can check for its compatibility.
You can see it used in graph/pull_v2_test.go.

php popen(exim) from a webpage

Problem: Apache accessing programs in /usr/bin
Platform:
Win/7 Pro Sp1
Apache 2.4
Php 5.6
Cygwin emulator gives Linux/Unix emu and has
Exim 4.84-1
program: sendit.php
hndl popen( exim ... )
fwrite(hndle, email_headers);
fwrite(hndle, email_body);
fflush(hndle);
fclose(hndle);
(a)launch from user account
php sendit.php -> exim/exim_main.log updated
reports 368 bytes sent to exim
-> email arrives
demonstrates sendit.php works & Exim config and permissions are correct
(b)launch from apache account uid(apache) gid(srvc)
php sendit.php -> exim/exim_main.log updated
reports 368 bytes sent to exim
-> email arrives
demonstrates sendit.php runs from any account
(c)launch from within a web page
Apache is run from its own uid/gid as in (b)
page.phtml sendit.php
-> NO update to exim log
-> no mail recv'd
reports same NNN bytes send to exim
NO I/O errors
I fwriting_stream 368 bytes ...
* wrote body fragment 368
* wrote total 368
I Wrote Body:(368) of 368
I Wrote EOT(5) of 5
I FLUSHING pipe
I closing pipe
I is closed pipe
I returning 3
adding -d+deliver+host_lookup+lookup+rewrite options to the popen(), yields nice trace - -
EXCEPT when run withing Apache - - nothing shows up
Apache cfg has ExecCGI in the directory and other php / perl programs run there.
Issue is Exim is not within the Apache environment (/usr/bin/exim)
So, created link /usr/bin/exim -> docroot/exim
and reference this instance popen(docroot/exim ...)
Still fails to deliver via pipe -> exim
Any ideas???
btw: have used popen in webpages several places with success
and yes, I verifed the popen return
$PIPE = popen( ... ...);if ($PIPE === false) die("*FE*);
STATUS: SOLVED 2015-05-30
Issue is Windows + Cygwin pathing
PHP tolerates Cygwin as $PATH is available.
Running within
Apache LoadModule php5_module "c:/php/php5.6/php5apache2_4.dll" does not
Must use an absolute WINDOWS path like
c:/cygwin/bin/exim-4.84-1.exe within the server.
php cmdPath_Tests.php
uses file_exists($path) exec to
find that path usable within Apache
Jeff#JeffPC7%
case 0 0 trying:
$ ls -l /usr/bin/exim
lrwxrwxrwx 1 Administrator None 24 Apr 6 09:19 /usr/bin/exim -> /usr/bin/exim-4.84-1.exe
Jeff#JeffPC7%
$ ls -l /usr/bin/exim /cygwin/bin/exim{,-4.84-1}.exe c:/cygwin/bin/exim-4.84-1.exe
case 1 1
ls: cannot access /cygwin/bin/exim.exe: No such file or directory
ls: cannot access /cygwin/bin/exim-4.84-1.exe: No such file or directory
lrwxrwxrwx 1 Administrator None 24 Apr 6 09:19 /usr/bin/exim -> /usr/bin/exim-4.84-1.exe
case 3 1 trying:
-rwxr-xr-x 3 Jeff None 1192467 Jan 25 19:16 c:/cygwin/bin/exim-4.84-1.exe
/usr/bin/exim
0 0 /usr/bin/exim
/usr/bin/exim-4.84-1.exe
0 1 /usr/bin/exim-4.84-1.exe
which: no exim-4.84-1.exe in (/cygwin/bin)
1 1 File Exists
cmd 1 1 trying: /cygwin/bin/exim-4.84-1.exe
results:
/usr/bin/exim-4.84-1.exe
2 1 trying: /usr/bin/exim-4.84-1.exe
I found:
3 1 trying: c:/cygwin/bin/exim-4.84-1.exe
cmd results:
2015-05-30 11:58:24 NP6FDA-000530-QD Completed
ie: email sent

Resources