I am trying to build a simple Suave.IO application to centralize the sending of emails. Currently the application has one endpoint that takes subject, body, recipients, attachments, and sender as form data and turns them into an EWS email message from a logging email account.
Everything works as intended in most cases, but I get a file corruption issue when one of the attachments is an excel file. In those cases, the file seems to get corrupted.
Currently, I am filtering the request.multipartFields down to only the ones that are marked as attachment files, and then doing this:
for (fileField: (string*string)) in fileFields do
let fname = (fst fileField)
let fpath = "uploadedFiles\\" + fname
File.WriteAllBytes(fpath, Encoding.ASCII.GetBytes (snd fileField)) |> ignore
The file path and the attachment names are then fed into the EWS message before sending.
Again, this seems to work with all attachments except attachments with binary. It seems like Suave.IO automatically encodes all multiPartFields as (string*string), which may require special handling when it's binary data.
How should I handle upload of binary files?
Thanks all in advance.
It looks like the issue was one of encoding. I was testing using python's request interface, and by default the files are encoded as multipart/form-data. By specifying a specific encoding for each file, I was able to help the server identify the incoming data as a file.
instead of
requests.post(url, data=data, files={filename: open(filepath, 'rb')})
I needed to make it
requests.post(url, data=data, files={filename: (filename, open(filepath, 'rb'), mimetypes.guess(filepath)})
With the second python script, files do end up in the files section of the request and I was able to save the excel file without corruption.
Related
I have a simple page in asp net 5, and users can upload their images there. Valid files are: *.jpg, *.png, so I'm doing steps below to validating the files:
Validating filename length : e.g: file name must be less than 50 alphabet characters
Validating filename : replacing any hidden or invalid characters
Validating file size : based on our configurations (e.g: less than 10MB)
Validating file extensions : based on our white-list: *.jpg, *.png
Validating Mimetypes : based on our white-list for IMAGE/JPEG, IMAGE/PNG
Validating file's first bytes (Magic Number) : based on our white-list for JPG: "FF-D8-FF-DB", "FF-D8-FF-E0" , "FF-D8-FF-EE" ,"FF-D8-FF-E1" and PNG: "89-50-4E-47"
Uploading the file with a random (guid) filename in the temp folder outside webroot: without any executing permissions.
Scanning the file with AV (Kaspersky or Norton Security) service installed.
But, some webshells can bypass these steps, like Insomnia webshell or others (they use the magic number headers at the first of file headers and inject their codes into some part of the file).
So my question is :
how can I detect and prevent webshell uploading?
Should I read and check the whole file for some black-list keywords?
Or what?
btw :We can't use any online webshell detection services.
This is a simple shell injected into a PNG file by woanware.co.uk:
I'm trying to read JSON-LD into Dask from Minio. The pipeline works but the strings come from Minio as binary strings
So
with oss.open('gleaner/summoned/repo/file.jsonld', 'rb') as f:
print(f.read())
results in
b'\n{\n "#context": "http://schema.org/",\n "#type": "Dataset",\n ...
I can simply convert this with
with oss.open('gleaner/summoned/repo/file.jsonld', 'rb') as f:
print(f.read().decode("utf-8"))
and now everything is as I expect it.
However, I am working with Dask and when reading into the a bag with
dgraphs = db.read_text('s3://bucket/prefa/prefb/*.jsonld',
storage_options={
"key": key,
"secret": secret,
"client_kwargs": {"endpoint_url":"https://example.org"}
}).map(json.loads)
I can not get the content coming from Minio to become strings vs binary strings. I need these converted before they hit the json.loads map I suspect.
I assume I can inject the "decode" in here somehow as well, but I can't resolve how.
Thanks
As the name implies, read_text opens the remote file in text mode, equivalent to open(..., 'rt'). The signature of read_text includes the various decoding arguments, such as UTF8 as the default encoding. You should not need to do anything else, but please post a specific error if you are having trouble, ideally with example file contents.
If your data isn't delimited by lines, read_text might not be right for you, and you can do something like
#dask.delayed()
def read_a_file(fn):
# or preferably open in text mode and json.load from the file
with oss.open('gleaner/summoned/repo/file.jsonld', 'rb') as f:
return json.loads(f.read().decode("utf-8"))
output = [read_a_file(f) for f in filenames]
and then you can create a bag or dataframe from this, as required.
I'm following the documentation to create a new upload session for a resumable file upload.
My request looks like:
/v1.0/me/drive/items/:folderId/children/:fileName/createUploadSession
This works when :fileName is something like test.txt or even test 2.txt. But throwing special characters in there like test".txt or test%22.txt cause the request to fail.
There no examples in the documentation on how to deal with special characters in this case, so is this supported?
File stored in OneDrive have similar naming conventions/restrictions to files stored locally. If you consider that OneDrive can sync to your local file system, it makes sense why this is the case.
In general, you should assume you cannot use any of these characters in your file names:
~ " # % & * : < > ? / \ { | }.
You can find the complete list at Invalid file names and file types in OneDrive, OneDrive for Business, and SharePoint.
I'd like to create a file object from an image located at a specific url. I'm downloading the file with Net Http:
img = Net::HTTP.get_response(URI.parse('https://prium-solutions.com/wp-content/uploads/2016/11/rails-1.png'))
file = File.read(img.body)
However, I get ArgumentError: string contains null byte when trying to read the file and store in into the file variable.
How can I do this without having to store it locally ?
Since File deals with reading from storage, it's really not applicable here. The read method is expecting you to hand it a location to read from, and you're passing in binary data.
If you have a situation where you need to interface with a library that expects an object that is streaming, you can wrap the string body in a StringIO object:
file = StringIO.new(img)
# you can now call file.read, file.seek, file.rewind, etc.
I have a script that's a plugin for redmine which enhances the application to send encrypted mail using gpg. At some point this stopped working. Unfortunately the one who wrote that script is not available anymore and I am an admin with only very limited knowledge of RoR.
The problem is, that obviously the script creates a file with the mail body, saves it to temp, encrypts it to an output file, reads this output and then sends the mail.
With an empty /tmp directory (such as after rebooting the whole server), the gpg.in file gets created when I try to send a test mail. But then I get an error that the gpg.out file was not available. Creating it using touch does cause an empty email being send so obviously the script does not write anything to that file.
File.open('/tmp/gpg.in', 'w') do |f1| #<--- Works, file is created
f1.puts(body)
end
list_of_keys = [ rec ]
Gpgr::Encrypt.file('/tmp/gpg.in', :to => '/tmp/gpg.out').encrypt_using(list_of_keys) #<- gpg.out wird nicht erzeugt.
text = ""
File.open('/tmp/gpg.out', 'r') do |f2| #<- throws file not found error, if file not there. When file was created empty using touch, it sends an empty mail
With my limited RoR knowledge, I can't figure out how to debug this. Permissions on /tmp are 777 so the script should be allowed to write there and obviously has because File.open('/tmp/gpg.in', 'w') works correctly all the time without an error. Hence I expect the problem in Gpgr::Encrypt.file not working correctly, but I also don't get any error from that function it fails silently.