In Data Integration module, I have created a mapping to load data from Csv file to Oracle table. I want to give a file pattern as the file name will have date in it. When I try to provide file pattern in the Source object, it is throwing the below error.
If someone can assist on letting me know how to load file with a file pattern, it will be very helpful.
Please let me know if you need any further details.
Try using a File Listener as the source for this mapping. In File Listener settings you can provide the pattern - and in turn, File Listener will trigger the mapping with the file found.
Related
Moving from nlog to serilog, I would like my .NET framework desktop application to reuse a statically-named log file each time I run it, but to clear out the contents of the file with each new process. Is it possible to configure serilog this way?
This is a similar question, but it's not quite the same. In the linked question, the user uses a new log file each time with a unique filename. In my case, I want to use the same log file name each time.
This is not something Serilog can do for you as of this writing.
Serilog.Sinks.File is hard-coded to open the file with FileMode.Append thus if the file already exists, it will always append contents at the end of the file.
FileLifecycleHooks allows you to intercept when the file is being opened, and that would give you an opportunity to remove the contents of the file (by calling SetLength(0) on the stream), but unfortunately the stream implementation that Serilog.Sinks.File uses (WriteCountingStream) does not support SetLength.
Your best bet is to just truncate or delete the log file yourself at the start of the app, and let Serilog create a new one.
e.g.
// Ensure that the log file is empty
using (var fs = File.OpenWrite("mylog.log")) { fs.SetLength(0); }
// ... Configure Serilog pipeline
I am running Neo4J on Docker within Vagrant.
I am attempting to LOAD CSV WITH HEADERS from a file within the /import/ directory (I had to move my file there) via a cURL request. My request looks something like this:
"LOAD CSV WITH HEADERS FROM \"file:///insert-neo4j.csv\" AS row ...
This provides me with the following error:
{"results":[],"errors [{"code":"Neo.ClientError.Statement.ExternalResourceFailed","message":"Couldn't load the external resource at: file:/var/lib/neo4j/import/insert-neo4j.csv"}]}
It is often suggested to me that I append the following to my '/conf/neo4j.conf' file, however this file DOES NOT EXIST, and creating it manually does not seem to work...
dbms.directories.import=import
dbms.security.allow_csv_import_from_file_urls=true
So I created the file /conf/neo4j.conf with the above variables, and I also tried adding these as environment variables to my docker-compose file. I seem to continuously have no luck uploading via CSV this way.
My questions are:
Is there anything blatantky wrong with this implementation?
Why does my /conf/neo4j.conf file NOT exist and how can I get it created?
Thank you
(p.s. my insert-neo4j.csv has -rwxr-xr-x)
The error message indicates it found the file but there is an error in the CSV ... most likely the formatting. Check this and if you can't see it, please post a few rows, including the header, of it so we might help.
I'm trying to get a setting from a configuration file (preferably something simple like .ini or JSON, not XML). If the file or setting does not exist, I want to be able to fall back to retrieving an environment variable.
I'd prefer to use an existing library for working with JSON/INI and not parsing the file myself. However, most libraries I've found won't work if a file doesn't exist.
How would I access a configuration value from a file that may or may not exist in F#?
You can use File.Exists to test whether or not the file exists:
open System.IO
let getConfig file =
if File.Exists file
then "config from file"
else "config from somewhere else"
OpenExeConfiguration (despite it's name) can open an arbitrary config file.
There's also the ASP.NET vNext Configuration stuff, outlined in this article which is quite flexible - no idea how separable (or relevant to your actual use case) it is [aside from the fact that you could conditionally include the config file into the config manager depending on whether it exists a la Mark's answer].
In addition to type providers, FSharp.Data provides some basic parsers, including JSON. This allows you to do a runtime check using File.Exists and then parse using your preferred utility.
I took the following approach in FAKE:
if File.Exists "local.json" then
let localVarProps = JsonValue.Parse(File.ReadAllText"local.json").Properties
for key, jsonValue in localVarProps do
setEnvironVar key (jsonValue.AsString())
I'm building an ASP.Net MVC4 application and the customer wants to be able to supply an XML configuration file, to configure a vendor list in the application, something like this:
<Vendor>
<Vendor name="ABC Computers" deliveryDays="10"/>
<Vendor name="XYZ Computers" deliveryDays="15"/>
</Vendors>
The file needs to be dropped onto a network location (i.e. not on the web server) and I don't have a database to import and store the data.
The customer also wants the ability to update it daily. So I'm thinking I'll have to do some kind of import (and validate the file) when the application starts up.
Any good ideas on the best way to accomplish this?
- The data needs to be quickly accessible
- Ideally I just want to import/store it once, or be able to access it quickly
- I need to be able to validate the file, so it might be prudent to be able to be able to switch to a backup
One thought was to use something like Entity Framework and simply read the file whenever I needed it, but if possible I'd hold it in memory in the application if possible.
Cheers
Vincent
No need to import it into a database or use Entity Framework. You can simply use .NET Xml Serialization to accomplish this.
The command line tool xsd.exe will generate c# classes from your Xml file. From the command line:
xsd.exe myfile.xml
xsd.exe /c myfile.xsd
The first command will infer and create an xml schema file (myfile.xsd) from your xml. The second command will convert the schema file to c# classes.
Then use the XmlSerializer class to deserialize your xml file into objects (assuming multiple objects in one file):
MyCollection myObjects= null;
string path = "mydata.xml";
XmlSerializer serializer = new XmlSerializer(typeof(MyCollection));
StreamReader reader = new StreamReader(path);
myObjects = (MyCollection)serializer.Deserialize(reader);
reader.Close();
You can use the .xsd file generated above to validate your xml files. Here's a link showing how: http://msdn.microsoft.com/en-us/library/ms162371.aspx.
When my Grails application starts, I build up a data structure from a CSV file downloaded from a remote URL. If the file is not accessible, I'd like to fall back to a local copy. Currently processing the file in the service layer, initiated using a Quartz job.
What is the best practice, using Groovy, for reading a local resource in Grails?
Where should I stash the file?
How do I safely and properly read the file?
General-case answers will be very acceptable.
I think the best way to deal with this is to store the file's location in an externalized configuration file.
So, you'd determine a standardized location (such as /etc/myappname/CSVFileConfig.groovy), or pass the config file path in using an environment variable or something similar. See Externalized Configuration for examples.
Then you can simply add the actual path to the local file to that extenal config, like so:
// CSVFileConfig.groovy
my.custom.csv.path = ...
Finally, access it using normal config operations:
// in your Quartz job
def path = grailsApplication.config.my?.custom?.csv?.path
if(!path) {
// no file to load
} else {
// load file
}
As far as reading the file, what are your primary concerns? If you are using a CSV library, such as OpenCSV (used in most of the Grails libraries for CSV parsing), it will handle the opening and parsing of the file.
For security issues beyond that, I'm not sure how to handle them in a generic way. It will depend on your specific scenario. I think the one coming from a URL has a higher risk factor.