Say that in a folder I have over 100+ .dat files with no common generic file names. For instance, the files are not named f001, f002, f003 (you get the pattern). The names are random. I need to parse these .dat files into SAS files. Each files have the same column/attributes. I use the following code to parse one of the .dat file:
data have;
infile 'C:\SAS\have.dat' dsd dlm='|';
input var1 var2 var3$;run;
The code is the same for each .dat files. Is there a way in SAS to simply parse all the files in a folder and name these SAS files the same as their original .dat. I want all the files to be separated and not under one SAS file.
[UPDATE]
I first start by reading all the filenames in my folder using the following SAS command:
data yfiles;
keep filename;
length fref $
8 filename $ 80;
rc = filename(fref,
'Y:\Files\Exchanges');
if rc = 0 then do ;
did = dopen(fref);
rc = filename(fref); end;
else do; length msg $200.;
msg = sysmsg();
put msg=; did =.;
end;
if did <=0 then putlog
'ERR' 'OR: Unable to open directory.';
dnum = dnum(did);
do i =1 to dnum;
filename = dread(did, i);
/* If this entry is a file, then output. */
fid = mopen(did, filename);
if fid >0 then output;
end;
rc = dclose(did);
run;
In yfiles I have all the names of my .dat datasets.
Now, how can I loop through each .dat files names of my yfiles dataset to apply the above parsing code?
Use CALL EXECUTE and a Data Step to loop through the file names. You use the Data Step to build and execute the SAS statements.
data _null_;
set yfiles;
format outStr $200.;
outStr = 'data have' || strip(put(_N_,best.)) || ';';
call execute(outStr);
outStr = "infile 'C:\SAS\" || strip(filename) || "' dsd dlm='|';";
call execute(outStr);
call execute("input var1 var2 var3$;run;");
run;
Try using a pipe file ref with the command doing a directory listing. Parse the output and loop over the directory contents.
Related
I'm using Lua in Scite on Windows, but hopefully this is a general Lua question.
Let's say I want to write a temporary string content to a temporary file in Lua - which I want to be eventually read by another program, - and I tried using io.tmpfile():
mytmpfile = assert( io.tmpfile() )
mytmpfile:write( MYTMPTEXT )
mytmpfile:seek("set", 0) -- back to start
print("mytmpfile" .. mytmpfile .. "<<<")
mytmpfile:close()
I like io.tmpfile() because it is noted in https://www.lua.org/pil/21.3.html :
The tmpfile function returns a handle for a temporary file, open in read/write mode. That file is automatically removed (deleted) when your program ends.
However, when I try to print mytmpfile, I get:
C:\Users\ME/sciteLuaFunctions.lua:956: attempt to concatenate a FILE* value (global 'mytmpfile')
>Lua: error occurred while processing command
I got the explanation for that here Re: path for io.tmpfile() ?:
how do I get the path used to generate the temp file created by io.tmpfile()
You can't. The whole point of tmpfile is to give you a file handle without
giving you the file name to avoid race conditions.
And indeed, on some OSes, the file has no name.
So, it will not be possible for me to use the filename of the tmpfile in a command line that should be ran by the OS, as in:
f = io.popen("python myprog.py " .. mytmpfile)
So my questions are:
Would it be somehow possible to specify this tmpfile file handle as the input argument for the externally ran program/script, say in io.popen - instead of using the (non-existing) tmpfile filename?
If above is not possible, what is the next best option (in terms of not having to maintain it, i.e. not having to remember to delete the file) for opening a temporary file in Lua?
You can get a temp filename with os.tmpname.
local n = os.tmpname()
local f = io.open(n, 'w+b')
f:write(....)
f:close()
os.remove(n)
If your purpose is sending some data to a python script, you can also use 'w' mode in popen.
--lua
local f = io.popen(prog, 'w')
f:write(....)
#python
import sys
data = sys.stdin.readline()
I am learning the Lua IO library. I'm having trouble with io.write(). In Programming Design in Lua, there is a piece of code that iterates through the file line by line and precedes each line with a serial number.
This is the file I`m working on:
test file: "iotest.txt"
This is my code
io.input("iotest.txt")
-- io.output("iotest.txt")
local count = 0
for line in io.lines() do
count=count+1
io.write(string.format("%6d ",count), line, "\n")
end
This is the result of the terminal display, but this result cannot be written to the file, whether I add IO. Output (" iotest.txt ") or not.
the results in terminal
This is the result of file, we can see there is no change
The result after code running
Just add io.flush() after your write operations to save the data to the file.
io.input("iotest.txt")
io.output("iotestout.txt")
local count = 0
for line in io.lines() do
count=count+1
io.write(string.format("%6d ",count), line, "\n")
end
io.flush()
io.close()
Refer to Lua 5.4 Reference Manual : 6.8 - Input and Output Facilities
io.flush() will save any written data to the output file which you set with io.output
See koyaanisqatsi's answer for the optional use of file handles. This becomes especially useful if you're working on multiple files at a time and gives you more control on how to interact with the file.
That said you should also have different files for input and output. You'll agree that it doesn't make sense to read and write from and to the same file alternatingly.
For writing to a file you need a file handle.
This handle comes from: io.open()
See: https://www.lua.org/manual/5.4/manual.html#6.8
A file handle has methods that acts on self.
Thats the function after the : at file handle.
So io.write() puts out on stdout and file:write() in a file.
Example function that can dump a defined function to a file...
fdump=function(func,path)
assert(type(func)=="function")
assert(type(path)=="string")
-- Get the file handle (file)
local file,err = io.open(path, "wb")
assert(file, err)
local chunk = string.dump(func,true)
file:write(chunk)
file:flush()
file:close()
return 'DONE'
end
Here are the methods, taken from io.stdin
close = function: 0x566032b0
seek = function: 0x566045f0
flush = function: 0x56603d10
setvbuf = function: 0x56604240
write = function: 0x56603e70
lines = function: 0x566040c0
read = function: 0x56603c90
This makes it able to use it directly like...
( Lua console: lua -i )
> do io.stdout:write('Input: ') local result=io.stdin:read() return result end
Input: d
d
You are trying to open the same file for reading and writing at the same time. You cannot do that.
There are two possible solutions:
Read from file X, iterate through it and write the result to another file Y.
Read the complete file X into memory, close file X, then delete file X, open the same filename for writing and write to it while iterating through the original file (in memory).
Otherwise, your approach is correct although file operations in Lua are more often done using io.open() and file handles instead of io.write() and io.read().
I think the tile explains it self how do I access these files and read and write them?
this is what my code looks like
local openFile = assert(io.open(wasd_ ,"r")) -- wasd_ is the .txt file I want to open
local t = openFile:read("r") -- read the file
I'm not sure if in the 1st line of code if I shut replace "wasd_" with "wasd_.txt" or the file path (i.e replace wasd_ with something like this --> C:/users/stuff/blah/thing/wasd_)
ant help is greatly appreciated
According to the documentation:
local Filename = "wasd_.txt"
local File = io.open(Filename, "r")
local FileContent = File:read("*all")
File:close()
The file will be open according to the current directory %CD%. If your current directory is C:\test\, then the open file will be C:\test\wasd_.txt. If you want to find another file, you could specify the full path C:\users\stuff\blah\thing\wasd_.txt
you can do it all-at-once if you want; Lua allows that, but you have more control over editing when you do it line-at-a-time; in case you need to use certain edits on certain lines. It's also much safer to write to a new file, so you still have your original, to compare and try again, if need be.
line by line, keeping original, writing to new text
local path = 'C:/users/stuff/blah/thing/wasd_.txt' -- wasd_ is the .txt file I want to open
local newpath = path :sub( 1, -5 ) ..'new.txt' -- C:/users/stuff/blah/thing/wasd_new.txt
local iput = assert( io.input( path ) )
local oput = assert( io.output( newpath ) )
local linenum = 0 -- used as a counter
while true do
linenum = linenum +1
local line = iput :read()
if line == nil then break end -- break out of loop if there's nothing else to read
local newline, _ = line :gsub( 'this', 'that' ) -- do your changes here
if linenum == 1 then newline = 'HEADER: ' ..newline end -- example of specific line edit
oput :write( linenum ..' ' ..newline ..'\n' ) -- read() line strips newline char, so add it back
end
iput :close()
oput :close()
You can take out all instances of linenum if it's not needed, but I would strongly recommend that you use two separate read & write files, at least until you're comfortable it's doing exactly what you want, and has no errors.
This is a question about Nifi.
I made Nifi pipeline to convert flowfile with xml format to csv format.
Now, I would like to concatenate or union the converted csv flowfile to existing tables by filename (which stands for table name as well).
Simply put, my processor flow is following.
GetFile (from a particular directory) -> 2. Convert xml to csv -> 3.Update the flowfile with table name
-> 4. PutFile (to a different directory)
But, at the end of the flow, PutFile processor throws an error, saying "file with the same name already exists".
I have no ideas how flowfile can be added to existing csv table.
Any advice, tips, ideas are appreciated.
Thank you in advance.
there is no support to append file however you could use ExecuteGroovyScript to do it:
def ff=session.get()
if(!ff)return
ff.read().withStream{s->
String path = "./out_folder/${ff.filename}"
//sync on file path to avoid conflict on same file writing (hope)
synchronized(path){
new File( path ).append(s)
}
}
REL_SUCCESS << ff
if you need to work with text (reader) content rather then byte (stream) content
the following example shows how to exclude 1 header line from flow file if destination file already exists
def ff=session.get()
if(!ff)return
ff.read().withReader("UTF-8"){r->
String path = "./.data/${ff.filename}"
//sync on file path to avoid conflict on same file writing (hope)
synchronized(path){
def fout = new File( path )
if(fout.exists())r.readLine() //skip 1 line (header) only if out file already exists
fout.append(r) //append to the file the rest of reader content
}
}
REL_SUCCESS << ff
I read an Excel 2003 file with a text editor to see some markup language.
When I open the file in Excel it displays incorrect characters. On inspection of the file I see that the encoding is Windows 1252 or some such. If I manually replace this with UTF-8, my file opens fine. Ok, so far so good, I can correct the thing manually.
Now the trick is that this file is generated automatically, that I need to process it automatically (no human interaction) with limited tools on my desktop (no perl or other scripting language).
Is there any simple way to open this XL file in VBA with the correct encoding (and ignore the encoding specified in the file)?
Note, Workbook.ReloadAs does not function for me, it bails out on error (and requires manual action as the file is already open).
Or is the only way to correct the file to go through some hoops? Either: text in, check line for encoding string, replace if required, write each line to new file...; or export to csv, then import from csv again with specific encoding, save as xls?
Any hints appreciated.
EDIT:
ADODB did not work for me (XL says user defined type, not defined).
I solved my problem with a workaround:
name2 = Replace(name, ".xls", ".txt")
Set wb = Workbooks.Open(name, True, True) ' open read-only
Set ws = wb.Worksheets(1)
ws.SaveAs FileName:=name2, FileFormat:=xlCSV
wb.Close False ' close workbook without saving changes
Set wb = Nothing ' free memory
Workbooks.OpenText FileName:=name2, _
Origin:=65001, _
DataType:=xlDelimited, _
Comma:=True
Well I think you can do it from another workbook. Add a reference to AcitiveX Data Objects, then add this sub:
Sub Encode(ByVal sPath$, Optional SetChar$ = "UTF-8")
Dim stream As ADODB.stream
Set stream = New ADODB.stream
With stream
.Open
.LoadFromFile sPath ' Loads a File
.Charset = SetChar ' sets stream encoding (UTF-8)
.SaveToFile sPath, adSaveCreateOverWrite
.Close
End With
Set stream = Nothing
Workbooks.Open sPath
End Sub
Then call this sub with the path to file with the off encoding.