Does data-flow-server and skipper have any active role after the deployment of the stream applications (other than maintaining state). Asked differently, say for example if I have an http source and amqp sink , does any of the traffic from http to amqp go through the data-flow-server or skipper?
Neither SCDF nor Skipper interact with or contribute towards streaming or batch processing. They are both responsible for the designing and the deployment of data pipelines made of streaming/batch applications. To do the designing and the deployment part, we provide a variety of tools, including UI, CLI, RESTful APIs, and Java DSL.
To say this differently, if you deploy a stream or a task data pipeline using SCDF, the applications involved in the data pipeline are solely responsible for the data processing—no data will ever go through SCDF or Skipper. The applications are independently addressing the use-case requirement. SCDF (and Skipper) will help with central monitoring, management, and continuous deliveries of the applications.
You can read more about the responsibilities from the architecture section in docs.
Is there any dashboard solutions available to communicate OPC Server directly (OLE for Process Control + Used in Industrial Standards)?
If so please suggest the dashboard vendor name.
I am using https://thingsboard.io/, an open-source IOT platform. The IOT Gateway service gathers the time-stamped data (telemetry) from OPC UA servers on my network. Thingsboard stores all the telemetry in a open-source, fault tolerant database, and provides web-based, customizable dashboards for visualization.
Check out the live demo.
I've not been doing bare metal TCP/IP for about 18 months, so I wonder what the current state of the art is.
I'm looking for both positive and negative aspects, with development of both server and client software.
I will be doing a project that needs a rock-solid TCP/IP layer, so for me that is an important aspect :)
For this to become a community wiki, I'm looking for broader answers than just 'rock solid'. So for instance information about the feature-width is also appreciated.
I'll be updating the question with relevant aspects found in the answers in order to get a wiki entry that has a balanced overview of those libraries.
For example, see my answer below with my past experience with Indy
I'm ambivalent on the exception handling and anti-freeze in Indy, though I got used to it, it still felt somewhat unnatural.
Right now I develop in both Delphi 2007 (non Unicode) and XE (Unicode), so the libraries I'm considering should support at least those two Delphi versions.
Edit: Summary of my past experience with Indy, and the comments (thanks Eugene, Marjan)
(please update with the current Indy state of the art):
Pro:
ships with Delphi
mature
development community
open source so lots of eyes scrutinizing those sources
a truckload of valuable comment documentation in the source code
OpenSSL support
supports a broad set of Delphi versions (including 2007 and XE)
wide choice of protocols
Con:
the version shipping with Delphi was not always the most stable one; download from the sources was usually required to get a stable build
(in the mean time) lots of duplication of code that now is in Delphi (but Indy requires for compatibility with older Delphi versions)
not all TCP/IP components were up-to-date (for instance, back then the POP3 client component did not support some basic POP3 commands)
version interoperability was a pain: upgrading from one Indy version to another could be very time consuming
I'm ambivalent on the exception handling and anti-freeze in Indy, though I got used to it, it still felt somewhat unnatural.
breaking changes are made between build updates; ifdefs required to accommodate those
Unclear release status if any at all, no RCs for a long while, getting trunk can make your local copy unstable
ICS - The Internet Component Suite
ICS - see www.overbyte.be. Open source by François Piette. To me this has always been the number 1 alternative to Indy. It's most interesting selling point: it makes using asynchronous stuff easy, and being async seems to be closer to "bare metal" sockets programming.
I've used it to build a fairly complex VNC proxy where the proxy itself (server) is built with ICS and the clients are a mixture of Indy and ICS. In periods of high demand the proxy handles about 100 simultaneous connections and about 10 simultaneous VNC screen sessions. It eats up an average of 5 mbit/s, handles connections over two distinct Internet connections. I don't think the 100 + 10 is the limit, because the server handles that without any problems and CPU usage is too low to mention.
Pros:
Works asynchronously
Somewhat easier on beginners because it doesn't need threads
Supports a good number of protocols
Cons:
Relies on Windows messaging. I'm simply not comfortable with that.
The async behavior makes implementing most protocols slightly difficult (because most protocols are in the form of send command / receive response). This shouldn't matter for most people since ICS offers ready-made implementation for the most-used protocols.
All that being said, I haven't used ICS in a very long time, I'm not up-to-date with all the bells and whistles. This is CW, so please edit and expand!
I have used Indy since 2003 for my own TCP communications framework. It is rock-solid, I have a version used with Delphi 2007 and another with Delphi 2010, if you handle the threadng correctly there is no need to use the anti-freeze stuff, and I have consistent exception handling on the client and the server by implementing my own wrapper around this.
You can dowload it here (http://www.csinnovations.com/framework_delphi.htm) - look for the Tcp units, mainly AppTcpServerUnt and AppTcpClientUnt.
I would strongly recommend Clever Internet Suite, it's by far the best designed and written set of communication components. It's not free and so not that well known, but it's well worth investigating.
Pro:
well designed and written
contains many components and implements various protocols.
supports a broad set of Delphi versions (including 2007 and XE)
SSL support
mature product as the release history indicates
Con:
not open source
You could consider using a higher protocol level like HTTP, because:
It's more firewall and VPN friendly;
It's well documented and known as a good protocol;
It already has secured HTTPS version;
It has a very low overhead over row TCP/IP;
It's ready to use in an AJAX environment (if you need it in the future);
Microsoft already did the low-level tuning for you in modern version of Windows.
In this case, you could take a look at two Open Source classes working from Delphi 6 up to XE:
THttpApiServer which implements a HTTP server using fast http.sys kernel-mode server:
The HTTP Server API enables applications to communicate over HTTP without
using Microsoft Internet Information Server (IIS). Applications can register
to receive HTTP requests for particular URLs, receive HTTP requests, and send
HTTP responses. The HTTP Server API includes SSL support so that applications
can exchange data over secure HTTP connections without IIS. It is also
designed to work with I/O completion ports.
The HTTP Server API is supported on Windows Server 2003 operating systems
and on Windows XP with Service Pack 2 (SP2). Be aware that Microsoft IIS 5
running on Windows XP with SP2 is not able to share port 80 with other HTTP
applications running simultaneously.
TWinHTTP which handles client-side HTTP/1.1 request using the WinHTTP API:
Microsoft Windows HTTP Services (WinHTTP) is targeted at middle-tier and
back-end server applications that require access to an HTTP client stack;
Is much faster than older WinINet API.
Resulting speed is very good (especially the server), and you will rely on Microsoft implementation. The first is the core of IIS, and the second is used in the latest versions of Internet Explorer.
The answer really depends on many factors and your requirements, such as
what layers are needed (TCP, SSL/TLS, application-level protocols)
whether you need a client or a server as well (server is much more complicated task)
whether you count paid options.
In general, not much (positive) happened in 18 months or even in 3 years as most developers look at .NET as primary development platform.
Clever Internet Suite mentioned in other answer and DevArt's SecureBridge gained some new functionality.
Our SecureBlackbox offers support for the most advanced features (besides native SSL/TLS): IPv6, HTTPS Proxy with basic, digest and NTLM authentication (starting with SecureBlackbox 9), International Domain Names (starting with SecureBlackbox 9), DNSSEC, bandwidth control and more.
Application-level protocols supported by SecureBlackbox are HTTP (client and server), WebDAV (client and server), FTP (client and server), SSH and SFTP (client and server), SMTP and POP3 clients, DNS client, AS2 and AS3. All of the protocols (besides SSH and SFTP, of course) have complete support for SSL/TLS.
The list of supported protocols can be found on Packages page. Supported protocol features are listed on Technical Specification page for each package.
Worked with NetMaster components way (way!) back in the old Delphi versions (2! 3! 4!)
Did some work with Indy, but had the unnatural feeling also (actually I'd describe it more as bulky)
Stumbled upon Synapse when I was searching for just a light wrapper around the Windows network API,
And then rediscovered plain old TTcpClient/TTcpServer. They are Delphi's own wrapper around winsock! I use them blocking, with a dedicated TThread inheritant for each TTcpClient, and let TTcpServer do the threads and do all the work in DoAccept, see here for an example.
This, fow now, gave me the rock-solid feel we're looking for. If you want to support heavy load, I would try and build a thread manager that handles several sockets/connections per thread, or have two sets of threads: a few that listen a larger number of 'dormant' connections, and the others that handle lesser 'active' connections, switching connections between the threads depending on wether a request or response is being handled. (e.g. HTTP's Connection: keep-alive)
As the question is - is a DBMS an example of a middleware technology?
Yes. "Middleware" can be defined as software that sits between an application program and the basic services of the Operating System. In general, Middleware abstracts those operating system services, and may provide additional functionality. A DBMS fits this definition perfectly, proving a natural interface for efficiently storing, organizing, and accessing data. The application program uses the DBMS API, rather than directly calling the operating system services for disk and/or memory access. COM and CORBA are also examples of Middleware.
"Communication Middleware" is a specific kind of middleware that abstracts the IPC services of the operating system. Communication Middlewares provide an API to perform inter-process communications (sending and receiving data), so the application program does not need to use the operating services to create and manage sockets, or pipes. Examples of Communication Middleware include DDS, JMS, and Corba.
No, a database system always is the backend part of an application's architecture.
An example for middleware would e.g. be COM or CORBA.
HTH.
Thomas
I'm looking for the best way to use a Windows Service to manage a group of objects for use by several different client programs. The service must
create the list of objects corresponding to physical or virtual devices,
maintain information about what process is currently using a given device and what devices are available to be used,
manage requests to use a given device by a process,
return devices to an unused state.
When I look at the Windows Service examples, though, there is very limited communications to Services. OnCustomCommand doesn't return any information to the caller.
So is there a recommended way to establish information exchange between a service and various clients?
For background - I am replacing a Win32/COM app with a .Net app. The current device manager is a COM server with the clients being COM clients.
You could always implement a WCF service inside your Windows Service to communicate with the outside world. It would do so using either HTTP or Net/TCP (in a corporate LAN environment) or other transports (like MSMQ message queueing, if appropriate).
When the Windows Service spins up, you could also bring up a WCF ServiceHost to handle those communication needs.
See:
MSDN: How to host a WCF service in managed Windows Service
Self-hosting WCF services
WCF Essentials - hosting the service
Just an option - MsgConnect library for transparent communications between applications on the same or different systems. Designed specifically with your task (service-GUI communication) in mind. With MsgConnect you can send and receive messages (much like you do in Windows) which carry custom data. On local system messages can be transferred using MMF or sockets, for remote communications TCP sockets or HTTP can be used.