I am testing batch data ingestion in Druid. Currently in my installation there is a folder $DRUID/var/tmp which grew to almost 1TB (terabyte, yes) now. And $DRUID/var/druid is mere 350GB. Is it safe to delete that $DRUID/var/tmp folder?
As was answered in Druid user group, it is safe to delete the contents of the tmp folder while no ingestion task is running.
Related
I lost the system tablespace and database control files permanently and there's no RMAN backup.
I only have a single database file copied from the docker container which had been destroyed permanently.
Is it possible to restore the data in the data file and how?
The only option I am aware of for your situation would be the use of Oracle's Data UnLoader (DUL) utility, which is only available by contracting their professional services consultants (not cheap). Short of that I don't believe there is any way to recover your data.
dbrecover for oracle can directly unload data from single datafile ,even you lost all other datafiles including system tablespace datafile . you can reference this video : https://youtu.be/iJNxy4WkA08
We have a dataflow application which reads from Pub/Sub, windows into fixed-size 1-minute duration windows and writes the raw messages to GCS with 10 shards. Our application has been running for 10 days now and it has created a .temp-beam-2017**** folder. There are about 6200 files under it and the count is growing every day.
My understanding is data flow will move the temp files to the specified output folder after the write is complete.
Could you please suggest what can be done in this case ? Each of these files are about 100MB.
inputCollection.apply("Windowing",
Window.<String>into(FixedWindows.of(ONE_MINUTE))
.triggering(AfterProcessingTime.pastFirstElementInPane()
.plusDelayOf(ONE_MINUTE))
.withAllowedLateness(ONE_HOUR)
.discardingFiredPanes()
)
//Writing to GCS
.apply(TextIO.write()
.withWindowedWrites()
.withNumShards(10)
.to(options.getOutputPath())
.withFilenamePolicy(
new
WindowedFileNames(options.getOutputPath())));
The discrepancy between 14400 and 13900 is most likely because your pipeline didn't get any data whose event time falls into a particular window and shard. When writing a windowed collection, we don't create empty files for "missing" windows, because in general, it is not possible to know which windows are "missing": it is, in theory, pretty clear for fixed or sliding windows, but not so for custom windowing functions or sessions etc. Moreover, assignment of shards is random, so it is possible that for a particular window very few data arrived, and then there's a pretty good chance that some of the 10 shards didn't get any of it.
As for why the temporary files are being left over: it seems that the pipeline is occasionally seeing exceptions when writing the files to GCS. The leftover files are "zombies" from those failed attempts to write data. We currently don't do a good job of cleaning up such files automatically in streaming mode (in batch, it is safe to delete the entire temporary directory when the pipeline is done, but in streaming we can't do that, and we delete only the individual files being renamed to their final location), but it should be safe for you to delete old temp files in that directory.
I've filed https://issues.apache.org/jira/browse/BEAM-3145 to improve the latter.
Let's say an app downloads images from web while the user is browsing the app. Let's assume there are virtually unlimited images and a new image is downloaded whenever the user demands one. These images are saved to tmp directory for caching purpose. Once the user closes the app, all the images downloaded are deleted by the app.
Now, as there are unlimited images, what will happen if the user requests next image, the storage is full and the app attempts to save the image to the tmp directory?
Will the previous images be deleted by the iOS automatically to provide the space required for the new images?
OR
Will the iOS start cleaning tmp directory associated with other apps?(If yes, what happens when the storage is full again and such cleaning has already taken place for all the other apps?)
OR
Will the app crash?
If you try and save a image to disk and the disk is full then NSData's
- (BOOL)writeToURL:(NSURL *)aURL
options:(NSDataWritingOptions)mask
error:(NSError **)errorPtr
Will return NO and an error object will be assigned to the errorPtr passed into the method. This error will have a NSFileWriteOutOfSpaceError. This error is very exceptional, and by the time you get it it's safe to say the system will have already notified the user that he is running out of disk space.
Having said that, a lot can be said about cleaning after yourself. If you're not going to use a saved image resource anymore then delete it from the file-system.
Cheers!
tmp/
Use this directory to write temporary files that do not need to persist between launches of your app. Your app should remove files from this directory when they are no longer needed; however, the system may purge this directory when your app is not running.
The contents of this directory are not backed up by iTunes.
That's the only thing documented. From this I can infer that it won't purge you tmp if your app is running,but it can purge the tmp of other apps which are not running
As #InderKumarRathore says, the docs imply that the system will not delete files from your temp directory when your app is running. The docs also don't promise that the system will delete contents from other apps' temp directories to make space for you.
I would suggest coding defensively: Keep track of the oldest/least recently used files in your temp directory and delete them yourself. Preflight file saves to make sure there is enough space, and display a message to the user if there isn't enough space to save the file(s).
I set the WinSCP temporary directory on my hard-drive, but after quitting WinSCP the files stored there get deleted.
Is it possible to prevent them from being deleted? So I can edit them or copy them later.
And if its possible, can WinSCP automatically load those files, if they are newer than the ones on the server? This is optional, but it would be good.
Is it possible to prevent them from being deleted?
Yes, the option is named Keep temporary copies of remote files in deterministic paths.
Can WinSCP automatically load those files, if they are newer than the ones on the server?
WinSCP has function Keep remote directory up to date that monitors local folder and automatically uploads any changes to the server.
If you run two instances of WinSCP, you can combine these two features, but it's quite strange setup.
In my application, we store temporary images generated from PDF in /tmp/pdf_images folder. Is it a standard practice? Or It's not recommended to use of /tmp from application code?
Assuming you're on linux or MacOS, this directory is just there for this.
Be sure to have them automatically deleted at closing of your application.
And note that they may be purged by the OS or the user (directly or not) at any moment when your application isn't running. And generally, it's totally cleaned at reboot (and every 3 days on MacOS).