how to upload files from a FTP location into Marklogic - upload

i need to upload files from a FTP location into marklogic. please guide me on this


MarkLogic doesn't allow accessing external FTP locations from XQuery, like it allows HTTP calls. Nor does it provide FTP servers, like it provides WebDAV servers.
You can however easily put a mediator in between that accesses the FTP instead, and use other means to upload the document into MarkLogic. The latter can be done through a WebDAV App Server that you can create using the Admin interface, through the built-in REST api in MarkLogic 6 ( http://docs.marklogic.com/REST ), or through custom code like Corona ( http://developer.marklogic.com/code/corona ).
If you write the mediator in Java, you can also use the Java API ( see Java API tab at http://docs.marklogic.com/ ).
HTH!


We have a app that needs documents from a shared folder that we running an etl on to get into marklogic. You can do this a number of ways. If you are able to I'd amount the drive on the marklogic box and then read from there. IF that doesnt work see if you can make those files viewable from an http-get requested. IF that doest work then you might want to make a web services.


I personally would avoid WebDav unless you absolutely need it.
Is this a one-off, batch , or continous job ?
If one-off or batch then I would suggest using a script to FTP the files to a local disk then using mlcp or RecordLoader or xmlsh to push them to MarkLogic.
If this is a continuous job then a custom Java app is probably the way to go.
Do realize that FTP is a horribly sensitive protocol .. it can fail in so many ways and takes special port openings etc. It was designed in the 80's before firewalls, NAT and such.
Getting FTP to work reliably irregardless of MarkLogic is a black magic art in itself.
If its possible to use another protocol then FTP that would be ideal.
Say scp or rsync or http.

Related

Access MP3 files on server from iOS.

I'm building a streaming app similar to pandora. However right now I'm storing all my files on http and accessing them with urls. Is there an alernative to this because all the files are in the public html folder? For example how does apps like pandora or spotify pull files off their servers. I'm new to web severs and not sure where to ask this question. I have a centos server on vps hosting with apache, MySQL, http, ftp.

You just need to provide the content as a bit stream rather than a file download. The source of that data to send as a stream can be stored as binary data in a BLOB column in a database or as a regular file on a non-public part of the file system. It really does not mater which one you use.
Storing them in the database gives your app a bit easier access and makes the app more portable since it is not restricted the file system level permissions.
The fact you currently have the files in a public folder is not really that critical of an issue since you are making them available for download. You would just need to make sure you have an authentication requirement if you want to restrict who can access them.

Access and write back to txt file on linux server from iOS on local network

I currently have a txt file on a Linux install that I need to access from my app and write back to. The app and the Linux server are on the same subnet and I have full control of both machines (permissions). I've thought about SSHing into the machine but this obviously has its security drawbacks sending raw credentials. Does anyone have any suggestions on what framework to use to create a secure tunnel or perhaps an alternative solution?

Write a simple web service in your language of choice php, python, ruby, etc. And make a simple secure call to your service with the changes.

Setting up a file server for integration with iOS apps

I need to set up a server so that files can be uploaded from an iOS app. I don't know how best to proceed. I thought about FTP but not sure if there is a better option.
any ideas appreciated
GC
Also I must add that I will be building the iOS app so can use server APIs in my code.

It's not ideal to set up a blind File/FTP server and hardcode the details into your app because all it takes is one person to intercept the login details and they have access to your server where they can upload (and potentially execute) bad things.
A possible idea could be to set up an API frontend on your server in a language of your choice (PHP, Ruby, Python or similar) where you can 'POST' images to the server. With the API frontend, you can also do validation to ensure that only valid images are being uploaded and all nefarious stuff is thrown away. Then, in your iOS app, you can set up code to interact with your API frontend and send the actual images which will then be stored on your server.
This is a fairly conceptual idea rather than an absolute implementation idea. It does require some thinking/reading and more setup/coding on the server side.
Edit: Just to add, if you only want a central location to store/get your images without controlling it on a per user basis then you may want to look into Amazon S3 as a File Server.

FTP Client/Server with Web Based UI/API

I don't even know where to begin to look for something like this, or whether or not one even exists. What I am generally looking for is a FTP Server that I can run in the background of any given linux distro. Particularly RedHat and Ubuntu. Having this server run as a service on the machine. Then through that I need a FTP client that can interact with that particular service thats built into a Browser based UI. It would be nice to have something that I can tie the server admin into the UI as well.
So my question here is, is there anything like that, that anyone knows of? The UI portion should be able to support (j)Ruby

F->It may fit the bill. It has a Javascript client wrapped in Rails on the server.
Read about it:
http://www.webresourcesdepot.com/web-based-ftp-client-similar-to-windows-explorer/
See a demo:
https://fit.jupiterit.com/
Download it:
http://code.google.com/p/f-it/

What is the standard method for a website to communicate with a win32 executable?

I have some delphi code which, given a list of items, calculates the total price taking into account any special deals that might apply.
This code is non-trivial to rewrite in another language.
How should I set it up to communicate with a website running on the same server? The website will need to ask it for a price every time the user updates their shopping cart. It's possible that there will be multiple concurrent requests.
The delphi code needs to maintain an in-memory list of special deals, periodically refreshed from a database. So it cannot simply be executed every time or anything as simple as that.
I don't know what the website is written in, or even which http server it runs under, so I'm just looking for ideas or standard methods.

It sounds like the win32 app is already running as a Windows Service on the box. So, if you can't modify that service, you are going to have to deal with whatever way it wants to accept and respond to requests. This could be through sockets or some higher level communication protocol like web services.
You could do a couple of things. Write an assembly that knows how to communicate with the service and have your web site use that assembly. Or you could build a shim service that knows how to communicate with the legacy service, but exposes communication over higher level protocols such as web services. Either way will have the benefit of hiding the concurrency, threading and communications issue behind an easy to call interface, but the latter will make communicating with the service easier for everyone going forward.

If you can modify the delphi app to take an XML request and respond with an XML answer over a TCP socket (ideally using the HTTP protocol), you will be able to make it interoperate with most web server frameworks relatively easily. But the exact details of how to make that integration happen will depend on the language/framework it was written in.
If the web server is on windows you can compile your delphi app as a DLL that can return XML or HTML, taking parameters as part of the URL or a POST operation. Some details on making a Delphi DLL for web servers are here.

It doesn't matter what web server or OS the existing system is running under. What matters is what you want YOUR code to run under. If it is windows then the easiest solution would be to use WebBroker and write a custom ISAPI application, or use SOAP to expose web services. The first method could be used if you wanted to write a rest like API for instance, the second if your web application has the ability to consume web services.
Another option, if you are running both on the same box under IIS, is to create a COM/Automation object which you then invoke via server side scripting (ASP). If the application is an ASP.NET application, then I would use PRISM to port your code into an assembly.

I have done this with a quite complicated workers compensation calculator. I created a windows service using RemObjects Sdk. The calculations are exposed as a soap method so it can be accessed by nearly anything.
It's not necessary to use RemObjects in the service but it makes it much easier to do as it handles a lot of the underlying plumbing. The clients don't need RemObjects, they just need to be able to call soap methods. Nearly any programming langugae can do that.
You could also create an isapi dll for IIS that exposes a soap interface. This would be useful if other websites on different servers needed access to the methods. However I have handled this in my case by opening a port in the firewall to access my windows service.
There is a lot of examples on the web. A couple of places to start reading are About.Com and Dr Bob.

Torn this app into Windows Service. Write Web Service that will communicate with your windows service. You should spend some time designing your Web Service, because this Web Service is going to be your consistent interface, shielding old Delphi app. So in the future whenever you will want to write web app, mobile app, or whatever you will imagine, you will have one consistent interface – XML Web Service.

A popular way to integrate a web application with background services is a message broker.
The message flow would be:
the web application sends a "calculation request" message to a message destination on the message broker, which contains all needed parameters and also a correlation id to match the calculation request with the response from the Delphi service
one (or, in a high availability / load balanced environment more) Delphi services handle the messages: pull the next incoming message, process it by feeding the parameters to the calculation engine, and send a "calculation result message" back to the web server
the web server can either synchronously wait for the response (and discard responses which have no matching correlation ide) and build the result HTML document, or continue with other tasks and asynchronously receive the calculation result in a separate thread, for example in a Ajax based web application
See for an introduction this slideshow about the Dopplr image service:
http://de.slideshare.net/carsonified/dopplr-its-made-of-messages-matt-biddulph-presentation

If you can make it a service (but not a library), you have to do inter-process communication somehow - there are a few ways to do this on Windows:
Sockets directly which is hardest since you have to do marshalling/auth yourself
Shared Memory (yuck!)
RPC which works great but isn't trivial
DCOM which is easier but a pain to configure
WCF - but can you call it from your Windows Service written in Delphi?

Resources