I am doing file integration using mirth. There is one software which generate the HL7 files. I want to read data from that files, without moving them to another destination. Next time when I want to read data, at that time it'll ignore the files from which the data are already read (i.e.Just read the new files data which are generated after last data read).
I had done this but I'll achieve it when I modify the original filename, if I am not modifying the filename then it'll read the duplicate data.
Is there is any solution for this problem, so we can read data from the files which are generated new. I am using mirth 3.5.1 version and HL7 v2 messages.
Thanks in advance.
Thanks #daveloyall, I am posting your comment as a answer here.
When you rename a file at the time you process it, for example, to add a .DONE suffix to the filename, you are adding information that can be used later. The part of the channel that reads files could be configured to skip files that have the .DONE suffix. You also add information if you move the files. Or store the filenames in some database table. I don't know if Mirth has an internal feature that tracks which HL7 messages it already processed, but if such a feature exists, the keyword 'deduplication' might be associated with it.
Related
I need to read csv files inside a folder. New csv files are generated every time a user submits a form. I'm using the "file" input plugin to read the data and send it to Influxdb. These steps are working fine.
The problem is that the same file is read multiple times every data collection interval. I was thinking of a solution where I could move the file that was read to a different folder, but I couldn't do that with Telegraf's "exec" output plug.
ps: I can't change the way csv files are generated.
Any ideas on how to avoid reading the same csv file multiple times?
As you discovered file input plugin is used to read entire files at each collection interval.
My suggestion is for you to instead use the directory monitor input plugin. This will read files in a directory, monitor the directory for new files, and parse the ones that have not already been picked up yet. There are some configuration settings in that plugin that make it easier to time when new files are read as well.
Another option is to use the tail input plugin which will tail a file and only read new updates to that file as things come. However, I think the directory monitor is more likely something you are after for your scenario.
Thanks!
I have some txt files that store some important data for my app. Due to its nature I want them to be in external text files. Currently i plan on reading them using a streamreader that reads the txt line by line. However, i don't know where to put my txt files, so i can access them in my streamreader which requires their path. Ive seen examples of using NSBundle.mainBundlepathforResource. However, I'm not really sure what a Bundle is or how to place my files there in the first place.
You can use NSBundle. Here's another answer that shows how to create and use bundles: https://stackoverflow.com/a/23884501/1228075
I am new to flume-NG. I want my source to send some unique xml files to the channel one by one. The channel will validate the xml files and send the validity(either true or false) and the xml file to th custom sink. This sink will write the valid files and invalid files to different directories in HDFS. I am not sure which source to use. Please help.
None of the current ones will fit your use case. The SpoolingDirectorySource is line-oriented so XML files will confuse it and not arrive in one piece.
I suggest you write a custom source for your application.
I got an app I'm working on that uses static data from a sqlite database to do various things, While I only need read only access to the database, depending on the episode they pick from the first screen I want it to use a different database file and I want the list of available episodes to be updateable on the fly. and I got help to get the list of available episodes updated, and the proper content downloaded and stored in separate folders, So I know I could when the episode is selected delete the sql file in the documents folder and copy in the new one each time and that would work well enough for what I'm trying to do. but it seems like a bit much extra work to have to check for file, delete file, copy in new one. then open it from there each time the user wants to pick a different episode. and I don't want to put all the sql files together as that will be a bigger hassle then the first route especially if this app stays around long enough to have a long list of episodes.
so my question here is: can I get at least read-only access to an sql file that I've downloaded (or one in the bundle for testing) with out having to first copy it to the documents? and if so how would i open the file?
Can I get at least read-only access to an SQL file that I've downloaded (or one in the bundle for testing) without having to first copy it to the documents directory?
Yes. Files in the app bundle are readable (if they weren't, there would be no point in storing files in the bundle).
And if so, how would I open the file?
It's not clear what you're asking here - if you want to perform SQL queries on the file, you should use the sqlite3 library which is available on iOS.
I would like to create a simple file repository in Ruby on Rails. Users have their accounts, and after one logs in they can upload a file or download files previously uploaded.
The issue here is the security. Files should be safe and not available to anyone but the owners.
Where, in which folder, should I store the files, to make them as safe as possible?
Does it make sense, to rename the uploaded files, store the names in a database and restore them when needed? This might help avoid name conflicts, though I'm not sure if it's a good idea.
Should the files be stored all in one folder, or should they be somewhat divided?
rename the files, for one reason, because you have no way to know if today's file "test" is supposed to replace last week's "test" or not (perhaps the user had them in different directories)
give each user their own directory, this prevents performance problems and makes it easy to migrate, archive, or delete a single user
put metadata in the database and files in the file system
look out for code injection via file name
This is an interesting question. Depending on the level of security you want to apply I would recommend the following:
Choose a folder that is only accessible by your app server (if you chose to store in the FS)
I would always recommend to rename the files to a random generated hash (or incremntally generated name like used in URL shorteners, see the open source implementation of rubyurl). However, I wouldn't store them in a database because filesystems are built for handling files, so let it do the job. You should store the meta data in the database to be able to set the right file name when the user downloads the file.
You should partition the files among multiple folders. This gives you multiple advantages. First, filesystems are not built to handle millions of files in a single folder. If you have operations that try to get all files from a folder this takes significantly more time. If you obfuscate the original file name you could create one directory for each letter in the filename and would get a fairly good distributed number of files per directory.
One last thing to consider is the possible collision of file names. A user should not be able to guess a filename from another user. So you might need some additional checks here.
Depending on the level of security you want to achieve you can apply more and more patterns.
Just don't save the files in the public folder and create a controller that will send the files.
How you want to organise from that point on is your choice. You could make a sub folder per user. There is no need to rename from a security point of view, but do try to cleanup the filename, spaces and non ascii characters make things harder.
For simple cases (where you don't want to distribute the file store):
Store the files in the tmp directory. DON'T store them in public. Then only expose these files via a route and controller where you do the authentication/authorisation checks.
I don't see any reason to rename the files; you can separate them out into sub directories based on the user ID. But if you want to allow the uploading of files with the same name then you may need to generate a unique hash or something for each file's name.
See above. You can partition them any way you see fit. But I would definitely recommend partitioning them and not lumping them in one directory.