I see there is no support for an FTP To Google Cloud Storage operator, only from SFTP to GCS.
I am trying to create a custom one using the following function, but I am getting a 404 error.
Is it possible to use the upload_from_file function using a temporary open file from the with statement?
with tempfile.NamedTemporaryFile(mode='rb') as file:
filename = file.name
self.ftp_hook.retrieve_file(
remote_full_path=remote_full_path, local_full_path_or_buffer=filename
)
file_path = os.path.join(destination_object)
self.log.info(file_path)
blob_file = gcs_bucket.blob(file_path)
blob_file.upload_from_file(file)
Related
I wants to read a excel file existing on Live URL of another website.
When I hit that URL in browser file is downloading. While in my rails app it is giving below error
No such file or directory # rb_sysopen - http://www.carsa.jp/admin/data.xlsx (Errno::ENOENT)
My Rails app code is as below
data = Roo::Excelx.new('http://www.carsa.jp/admin/data.xlsx')
header = data.row(1)
puts header
Note: If I download file and place it within my application it is working fine but the requirement is to read it from the third-party website in a scheduled job as per the above script.
data = Roo::Excelx.new('lib/data.xlsx')
header = data.row(1)
puts header
Try using Roo::Spreadsheet.open instead of Roo::Excelx.new. According to the Roo Readme:
Roo::Spreadsheet.open can accept both paths and File instances.
This should do the trick:
Roo::Spreadsheet.open('http://www.carsa.jp/admin/data.xlsx')
I followed this page:
https://cloud.google.com/speech/docs/getting-started
and I could reach the end of it without problems.
In the example though, the file
'uri':'gs://cloud-samples-tests/speech/brooklyn.flac'
is processed.
What if I want to process a local file? In case this is not possible, how can I upload my .flac via command line?
Thanks
You're now able to process a local file by specifying a local path instead of the google storage one:
gcloud ml speech recognize '/Users/xxx/cloud-samples-tests/speech/brooklyn.flac' \ --language-code='en-US'
You can send this command by using the gcloud tool (https://cloud.google.com/speech-to-text/docs/quickstart-gcloud).
Solution found:
I created my own bucket (my_bucket_test), and I upload the file there via:
gsutil cp speech.flac gs://my_bucket_test
If you don't want to create a bucket (costs extra time and money) - you can stream the local files. The following code is copied directly from the Google cloud docs:
def transcribe_streaming(stream_file):
"""Streams transcription of the given audio file."""
import io
from google.cloud import speech
client = speech.SpeechClient()
with io.open(stream_file, "rb") as audio_file:
content = audio_file.read()
# In practice, stream should be a generator yielding chunks of audio data.
stream = [content]
requests = (
speech.StreamingRecognizeRequest(audio_content=chunk) for chunk in stream
)
config = speech.RecognitionConfig(
encoding=speech.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=16000,
language_code="en-US",
)
streaming_config = speech.StreamingRecognitionConfig(config=config)
# streaming_recognize returns a generator.
responses = client.streaming_recognize(
config=streaming_config,
requests=requests,
)
for response in responses:
# Once the transcription has settled, the first result will contain the
# is_final result. The other results will be for subsequent portions of
# the audio.
for result in response.results:
print("Finished: {}".format(result.is_final))
print("Stability: {}".format(result.stability))
alternatives = result.alternatives
# The alternatives are ordered from most likely to least.
for alternative in alternatives:
print("Confidence: {}".format(alternative.confidence))
print(u"Transcript: {}".format(alternative.transcript))
Here is the URL incase the package's function names are edited over time: https://cloud.google.com/speech-to-text/docs/streaming-recognize
I'm working on a Lua script which will be hosted by a third party program (some .exe which will call a certain function in my script). In order to implement a functionality I need (make a rest call to a webservice to retrieve certain info) I want to use socket.http.request.
I've first build an example script for the call I wanted to make:
local io = require("io")
local http = require("socket.http")
local ltn12 = require("ltn12")
local data = "some data")
local response = {}
socket.http.request({
method = "POST",
url = "http://localhost:8080/someServce/rest/commands/someCommand",
headers = {
["Content-Type"] = "application/x-www-form-urlencoded",
["Content-Length"] = string.len(data)
},
source = ltn12.source.string(data),
sink = ltn12.sink.table(response)
})
print(table.concat(response))
print("Done")
This works fine. I get the response I expect.
Now when I try to do this from the third party host, I first got an error:
module 'socket.http' not found:
no field package.preload['socket.http']
no file '.\socket\http.lua'
no file 'D:\SomeFolder\lua\socket\http.lua'
no file 'D:\SomeFolder\lua\socket\http\init.lua'
no file 'D:\SomeFolder\socket\http.lua'
no file 'D:\SomeFolder\socket\http\init.lua'
no file 'C:\Program Files (x86)\Lua\5.1\lua\socket\http.luac'
no file '.\socket\http.dll'
no file 'D:\SomeFolder\socket\http.dll'
no file 'D:\SomeFolder\loadall.dll'
no file '.\socket.dll'
no file 'D:\SomeFolder\socket.dll'
no file 'D:\SomeFolder\loadall.dll'
I've tried copying the socket folder from the LUA folder to the folder the host is executing from (D:\SomeFolder). It then finds the module, but fails to load it with another error:
loop or previous error loading module 'socket.http'
I've also tried moving the require statement outside of the function and making it global. This gives me yet another error:
module 'socket.core' not found:
no field package.preload['socket.core']
no file '.\socket\core.lua'
no file 'D:\SomeFolder\lua\socket\core.lua'
no file 'D:\SomeFolder\lua\socket\core\init.lua'
no file 'D:\SomeFolder\socket\core.lua'
no file 'D:\SomeFolder\socket\core\init.lua'
no file 'C:\Program Files (x86)\Lua\5.1\lua\socket\core.luac'
no file 'C:\Program Files (x86)\Lua\5.1\lua\socket\core.lua'
no file '.\socket\core.dll'
no file 'D:\SomeFolder\socket\core.dll'
no file 'D:\SomeFolder\loadall.dll'
no file '.\socket.dll'
no file 'D:\SomeFolder\socket.dll'
no file 'D:\SomeFolder\loadall.dll'
Then I tried copying the core.dll from socket into the D:\SomeFolder folder and it gave me another error:
error loading module 'socket.core' from file '.\socket\core.dll':
%1 is not a valid Win32 application.
Now I'm stuck. I think I must be doing something completely wrong, but I can't find any proper description on how to fix issues like this. Can anyone help me out?
As it turns out, the actual path Lua is going to look for is the problem here. Together with the third party we found that if we put a set of libraries in D:\SomeFolder\ everything now works. So for example there is now a socket.lua in D:\SomeFolder\and there are a socket and a mime forlder there as well.
Rule of thumb appears to be that the location of lua5.1.dll that is bound by the application is leading for the location of any modules you want to load.
You probably need to have the following folder structure (relative to your D:\SomeFolder folder):
socket.lua
socket/core.dll
socket/http.lua
socket/url.lua
socket/<any other file from socket folder required by http.lua>
I just tested this configuration and it works for me.
loop or previous error loading module 'socket.http'
This is usually caused by loading socket.http from socket/http.lua file itself.
I'm trying to save an image from an OpenLDAP server. It's in binary format and all my code appears to work, however, the image is corrupted.
I then attempted to do this in PHP and was successful, but I'd like to do it in a Grails project.
PHP Example (works)
<?php
$conn = ldap_connect('ldap.example.com') or die("Could not connect.\n");
ldap_set_option($conn, LDAP_OPT_PROTOCOL_VERSION, 3);
$dn = 'ou=People,o=Acme';
$ldap_rs = ldap_bind($conn) or die("Can't bind to LDAP");
$res = ldap_search($conn,$dn,"someID=123456789");
$info = ldap_get_entries($conn, $res);
$entry = ldap_first_entry($conn, $res);
$jpeg_data = ldap_get_values_len( $conn, $entry, "someimage-jpeg");
$jpeg_filename = '/tmp/' . basename( tempnam ('.', 'djp') );
$outjpeg = fopen($jpeg_filename, "wb");
fwrite($outjpeg, $jpeg_data[0]);
fclose ($outjpeg);
copy ($jpeg_filename, '/some/dir/test.jpg');
unlink($jpeg_filename);
?>
Groovy Example (does not work)
def ldap = org.apache.directory.groovyldap.LDAP.newInstance('ldap://ldap.example.com/ou=People,o=Acme')
ldap.eachEntry (filter: 'someID=123456789') { entry ->
new File('/Some/dir/123456789.jpg').withOutputStream {
it.write entry.get('someimage-jpeg').getBytes() // File is created, but image is corrupted (size also doesn't match the PHP version)
}
}
How would I tell the Apache LDAP library that "image-jpeg" is actually binary and not a String? Is there a better simple library available to read binary data from an LDAP server? From looking at the Apache mailing list, someone else had a similar issue, but I couldn't find a resolution in the thread.
Technology Stack
Grails 2.2.1
Apache LDAP API 1.0.0 M16
Have you checked whether the image attribute value is base-64 encoded?
I found the answer. The Apache Groovy LDAP library uses JNDI under the hood. When using JNDI certain entries are automatically read as binary, but if your LDAP server uses a custom name, the library will not know that it's binary.
For those people that come across this problem using Grails, here's the steps to set a specific entry to binary format.
Create a new properties file call "jndi.properties" and add it to your grails-app/conf directory (all property files in this folder are automatically included in the classpath)
Add a line in the properties file with the name of the image variable:
java.naming.ldap.attributes.binary=some_custom_image
Save the file and run the Grails application
Here is some sample code to save a binary entry to a file.
def ldap = LDAP.newInstance('ldap://some.server.com/ou=People,o=Acme')
ldap.eachEntry (filter: 'id=1234567') { entry ->
new File('/var/dir/something.jpg').withOutputStream {
it.write entry.image
}
}
I'm new with lua but working on an application that works on specific files with given path. Now, I want to work on files that I download. Is there any lua libraries or line of codes that I can use for downloading and storing it on my computer ?
You can use the LuaSocket library and its http.request function to download using HTTP from an URL.
The function has two flavors:
Simple call: http.request('http://stackoverflow.com')
Advanced call: http.request { url = 'http://stackoverflow.com', ... }
The simple call returns 4 values - the entire content of the URL in a string, HTTP response code, headers and response line. You can then save the content to a file using the io library.
The advanced call allows you to set several parameters like HTTP method and headers. An important parameter is sink. It represents a LTN12-style sink. For storing to file, you can use sink.file:
local file = ltn12.sink.file(io.open('stackoverflow', 'w'))
http.request {
url = 'http://stackoverflow.com',
sink = file,
}