Premise:
I have a windows service, which acts as a client to an IIS-based ASP.NET MVC3 application. They are both running on the same machine, and I'm trying to make them communicate with each other through SignalR.
I'm using the SignalR.Client library downloaded from Nuget on the windows service, and I have successfully connected it to the IIS. However, now that I'm calling a function from the MVC side SignalR Hub, which should propagate to the clients (including my Windows Service) - the Windows Service is not receiving any events.
What I've tried:
I've tried subscribing to the server side events in many ways but here is my current code:
private static Connection WebUIConnection = new HubConnection(ConfigurationManager.AppSettings["localIISLocation"]);
private static IHubProxy webUiHubProxy = new HubProxy(WebUIConnection, "Hub");
----
WebUIConnection.Credentials = new System.Net.NetworkCredential(userName, password);
webUiHubProxy.On("invoke", () => Console.WriteLine("Derp"));
WebUIConnection.Start().Wait();
On the IIS side I then call:
Clients.invoke();
So, I've tried invoking things from the windows service, and those work perfectly, however - when the information needs to go from IIS-> Windows Service, absolutely nothing happens.
Now, seeing as the SignalR has dynamic binding going inside it, I don't see any exceptions being thrown and I have absolutely no idea how to approach debugging this. I've checked the application logs with Event Viewer, and there is nothing inconclusive there.
I'm thinking, if maybe there is a system in place in Windows which disallows IIS to connect to the local machine (as a security measure), but it is weird that the connection is working perfectly in one direction and not the other.
Any ideas on how to debug this, or in fact what this could be about?
Best regards,
Lari
Related
Error message in Application Insights:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (OurApiUrlAddress:443) A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
It's always a 21 seconds TCP timeout, this is a very generic error I know, but the reason for this error is not always the same, I've been reading all the threads about this. We've been investigating this problem for months with no luck, we're also in contact with Azure team.
Important: this same site written in RUBY was using this same API without any problem in the past, the API is responsive and it's called from other sites without any problem, but this specific site was migrated from RUBY to .NET and at the same time this site was hosted in AZURE, this are the 2 big changes. This just happens when the site (remember it's hosted in Azure) calls to API / services hosted in our company, this doesn't happen when site calls a service hosted somewhere else, these makes us think the problem may be related to the company infrastructure but it can't be that alone, this has to be related to .NET and AZURE someway since these APIs and services respond perfectly to calls from other sites hosted in our network and they were working fine with the ruby version of this site. These Apis and services are not throwing this error when called in-browser from outside the company network.
The services/apis are behind a firewall but ports are perfectly configured (there are not any other traffic apps nor devices at play).
This error doesn't seem to be related to port exhaustion or SNAT, since sometimes only 1 developer alone is working in the DEV environment and he gets this socket exception error.
Just to give an idea we're getting around 250 socket exceptions a day on production, and this is just a small percentage of all the calls, so there is something that, just sometimes, is making this happen.
We know about the well known HttpClient issue when multiple instances are created, so we decided to use the Singleton approach ensuring only 1 instance per API/Service, as I'll show here, this is the call that gives more socket exceptions:
In StartUp class/file:
services.AddSingleton<IUploadApi>(new UploadApi(new HttpClient() { BaseAddress = new Uri(appSettings.Endpoints.UploadServicesUrl) }));
Part of appsettings.json:
"Endpoints": {
"UploadServicesUrl": "https://ourApiUrlAddress"
},
UploadApi.cs
public interface IUploadApi
{
Task<UploadArtworkViewModel.UploadConfigurationData> GetUploadConfiguration();
}
public class UploadApi : IUploadApi
{
private readonly HttpClient httpClient;
public UploadApi(HttpClient client)
{
httpClient = client;
}
public async Task<UploadArtworkViewModel.UploadConfigurationData> GetUploadConfiguration()
{
var response = await httpClient.GetAsync("api/GetUploadConfiguration").ConfigureAwait(false);
var json = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
return JsonConvert.DeserializeObject<UploadArtworkViewModel.UploadConfigurationData>(json);
}
}
Call from controller:
model.UploadConfiguration = await UploadApi.GetUploadConfiguration().ConfigureAwait(false);
Any idea on things to test or places to look are welcome, obviously I've not been able to reproduce this one. We know there's always a 21 seconds timeout, that's a TCP timeout, but that doesn't help much. Maybe for some reason the connection is dropped or Azure is having problems (sometimes) when accessing the company network. I can post more info from application insights if needed but I don't see anything special there about the error.
EDIT - More info: It happens when any API or service is called from this MVC site Controllers, so the problem appears sporadically (still like 300 times per day) when the site server tries to reach an API or service, this makes me believe it's something related to the company infraestructure, but still no idea what it could be.
From asp.net monsters:
"the application has exited and yet there are still a bunch of these
connections open"
"They are in the TIME_WAIT state which means that the connection has
been closed on one side (ours) but we’re still waiting to see if any
additional packets come in on it because they might have been delayed
on the network somewhere."
Even if you're using a singleton HttpClient, it seems that some of the connections are awaiting for additional packages which leads to socket exaustion.
The solution is to change your code and use HttpClientFactory or HttpClientFacotoryLite. The reason to use HttpClientFactory is that produces HttpClient instances that resuse Socket handlers from a pool of socket handlers. The handlers are recycled periodically to also take care of DNS changes. In summary, when using HttpClientFactory, HttpClient delegates work to a SocketClientHandler.
We finally got this problem fixed after working together with Azure team for some time, it was a gateway problem, solution was applying NAT/Vnet Integration. This is what we did to fix it:
https://learn.microsoft.com/en-us/azure/app-service/networking/nat-gateway-integration
We've got a mirrored SQL server 2012 database setup on Azure VM's - two servers plus a witness, all using client certificates, with SQL logins with the same SID set.
When testing our app from a different VM, everything works as expected when we manually failover the database, there's a one second wait and then it continues to operate quite happily.
If we then do another manual failover, ie moving the principal back to the original server, the app errors and throws a 'no such host in known' error. Recycling the app pool fixes the issue, but this clearly isn't workable in production when one of the servers is updated followed by the other at some later point (both are in an availability set).
The host not known error is somewhat baffling as it was communicating with it happily before the initial failover, and will again after the app pool recycle.
Here's the connection string as it is right now, after a lot of faffing around:
"Data Source=server1,1433;Failover Partner=server2,1433;Initial
Catalog=;MultipleActiveResultSets=True;User Id=user;
Password=password; Network=dbmssocn;Connect Timeout=60; async = true;"
providerName="System.Data.SqlClient"
The app is running on .net 4.5.2, so should be up to date with hotfixes, and we're out of ideas after much Googling with Bing.
I've just solved a problem that I had that looks very similar to your problem. I'd get the host not known error whenever the database switched from the first one listed in the web.config file to the failover one. It was fine switching from the failover to the primary.
The problem that I had was that I set up the database mirroring using server names but my web server did not know the database servers by name. Once I fixed this, I was able to get the failover working smoothly both ways.
This is what I think was happening:
I set up the mirroring using the names SQL1 and SQL2 as the principal and mirror servers
I have their ip addresses in my connection string: 10.1.1.5 and 10.1.1.6
The application tries to get to the first server 10.1.1.5 and succeeds and is then told that the mirror server is SQL2
SQL1 goes down and the database is successfully switched to the mirror server.
The web application attempts to connect, fails and determines that it should try the second server.
It tries to connect to SQL2, which it doesn't know, and fails with the message that the host is unknown.
This answer would only apply to your situation if you actually put ip addresses in your web.config and that server1,1433 and server2,1433 were actually masking place-holders for the ip addresses that you actually used.
I haven't really solved the naming issue though. I just added the two database server names to the HOSTS file which isn't an acceptable situation but does prove my theory on what my problem was.
I am researching a setup just like you have and upon reading this and the response by Steve Kaye, I'm wondering if you have SQL browser running. Take a look at this article for how SQL browser comes into play:
http://blogs.msdn.com/b/spike/archive/2010/12/15/running-a-database-mirror-setup-with-the-sqlbrowser-service-off-may-produce-unexpected-results.aspx
Premise
I'm attempting to make a windows service act as a signalR client to a web server (MVC3 project running on IIS Express). When trying to connect to the server, a 401 Unauthorized is returned.
Now, as far as I understand, the windows service runs under the account NETWORK_SERVICE, and it makes sense that this is not a valid user name to connect to the IIS. However, I've tried configuring SignalR in the following way:
Init
private static Connection WebUIConnection = new Connection("http://localhost:54193/IISWebsite");
Set credentials
WebUIConnection.Credentials = new System.Net.NetworkCredential("?", "?")
The settings of the IIS:
The IIS is almost a standard MVC3 project, and it has windows authentication enabled.
What I've tried
I've tried setting SignalR's credentials as my local windows username + pw, and also tried using the local network AD uname + pw, but I don't see this as being the way to do it.
So what I'm asking is, what should I consider when I try to make my windows service act as a client to the signalR-server, and is there a way to configure IIS to give client access to the Network_Service user? Is it in fact possible to make a windows service act as a client to a web server running in IIS like I'm trying to do?
Thanks,
Lari
I was able to solve what seems to be the problem. With the SignalR.Client the URL you need to give in the connection string looks like this:
private static Connection WebUIConnection = new Connection("http://localhost:54193/IISWebsite/signalr/hubs");
In addition to this I had to provide it with my local admin username and password for it to be able to connect. This is not how it should be done I feel, and I'd appreciate if anyone can enlighten me on how to make the network user have access to IIS from a local machine.
However, I think making a separate administrator account and making my windows service run under that account in the production environment is a solution I'm willing to accept. This way, it will be able to connect to the local IIS without a hitch.
Hope this information is helpful to others :)
Lari
If you're using hubs then that code is incorrect. There's a HubConnection you should be using. More info here:
https://github.com/SignalR/SignalR/wiki/SignalR-Client-Hubs
I have created a WCF service using the NetMsmq binding for which i created a private queue on my machine and executed the project. This works fine as such and my WCF service is started and accesses the message using the queue in the debugging environment. Now, I wanted to host the service using the windows service and for the same I created a new project and windows installer as well (This service runs under Local System Account). Then I tried installing this windows service using the InstallUtil command through the command prompt. When installation is happening and during the service host opening, I get an exception saying:
There was an error opening the queue. Ensure that MSMQ is installed and running, the queue exists and has proper authorization to be read from. The inner exception may contain additional information.
Inner Exception System.ServiceModel.MsmqException: An error occurred while opening the queue:Access is denied. (-1072824283, 0xc00e0025). The message cannot be sent or received from the queue. Ensure that MSMQ is installed and running. Also ensure that the queue is available to open with the required access mode and authorization.
at System.ServiceModel.Channels.MsmqQueue.OpenQueue()
at System.ServiceModel.Channels.MsmqQueue.GetHandle()
at System.ServiceModel.Channels.MsmqQueue.SupportsAccessMode(String formatName, Int32 accessType, MsmqException& msmqException)
Could anyone suggest the possible solution for the above issue? Am I missing any permissions to be set for the queue as well as the windows service, if so could you suggest where should these permissions be added?
Tom Hollander had a great three-part blog series on using MSMQ from WCF - well worth checking out!
MSMQ, WCF and IIS: Getting them to play nice (Part 1)
MSMQ, WCF and IIS: Getting them to play nice (Part 2)
MSMQ, WCF and IIS: Getting them to play nice (Part 3)
Maybe you'll find the solution to your problem mentioned somewhere!
Yes, it looks like a permissions issue.
Right click on your private queue from the Server Manager, and select Properties. Proceed to the Security tab, and make sure you have the right permissions in there for your Local System Account.
This is also confirmed in Nicholas Allen's article: Diagnosing Common Queue Errors, where the author defines the error code 0xC00E0025 as a permissions problem.
I ran into same problem, here is the solution.
Right click "My Computer" --> Manage. In Computer Management window go to "Services and Applications --> Message Queueing --> ur queue", select ur queue and access properties. Add the user running ur WCF application and give full access. This should solve the issue.
Can simple be that the service can't find the it's queue.
The queue name must exact match the endpoint address.
Example:
net.msmq://localhost/private/wf.listener_srv/service.svc
points to local queue
private$\wf.listener_srv\service.svc
If queue name and endpoint are according to each other, then is most like that the credentials defined on the IIS pool don't grant access to the queue.
We use stateful DataSnap servers for some business logic tasks and also to provide clientdataset data.
If we have to update the server to modify a business rule, we copy the new version into a new empty folder and register it (depending on the Delphi version, just by launching or by running the TRegSvr utility).
We can do this even while the old server instance is running. However, after registering the new version, all new client connections will still use the currently running (old) server instance. All clients have to disconnect first, then the new server will be used for the next clients.
Is there a way to direct all new client connections to the new server, immediately after registering?
(I know that new or changed method signatures will also require a change and restart of the clients but this question is about internal modifications which do not affect the interface)
We are using Socket connections, and all clients share the same server application (only one application window is open). In the early days we have used a different configuration of the remote datamodule which resulted in one app window per client. Maybe this could be a solution? (because every new client will launch the currently registered executable)
Update: does Delphi XE offer some support for 'hot deployment' (of updated servers)? We use Delphi 2009 at the moment but would upgrade to XE if it offers easier implementation of 'hot deployment'.
you could separate your appserver into 2 new servers, one being a simple proxy object redirecting all methods (and optionally containing state info if any) to the second one actually implementing your business logic. you also need to implement "silent reconnect" feature within your proxy server in order not to disturb connected clients if you decide to replace business appserver any time you want. never did such design myself before but hope the idea is clear
Have you tried renaming the current server and placing the new in the same location with the correct name (versus changing the registry location). I have done this for COM libraries before with success. I am not sure if it would apply to remote launch rules through as it may look for an existing instance to attach to instead of a completely fresh server.
It may be a bit hackish but you would have the client call a method on the server indicating that a newer version is available. This would allow it to perform any necessary cleanup so it doesn't end up talking to both the existing server instance and new server instance at the same time.
There is probably not a simple answer to this question, and I suspect that you will have to modify the client. The simplest solution I can think of is to have a flag (a property or an out parameter on some commonly called method) on the server that the client checks periodically that tells the client to disconnect and reconnect (called something like ImBeingRetired).
It's also possible to write callbacks under certain circumstances for datasnap (although I've never done this). This would allow the server to inform the client that it should restart or reconnect.
The last option I can think of (that hasn't already been mentioned) would be to make the client/server stateless, so that every time the client wants something it connects, gets what it wants then disconnects.
Unfortunately none of these options are the answer you want to your question, but might give you some ideas.
(optional) set up vmware vSphere, ESX, or find a hosting service that already has one.
Store the session variables in db.
Prepare 2 web boxes with 2 distinct IP address and deploy your stuff.
Set up DNS, firewall, load balancer, or BSD vm so name "example.com" resolves to web box 1.
Deploy new version to web box 2.
Switch over to web box 2 using whatever routing method you chose.
Deploy new version to web box 1 if things look ok.
Using DNS is probably easiest, but it takes time for the mapping to propagate to the client (if the client is outside your LAN) and also two clients may see different results. Some firewalls have IP address mapping feature that you can map public IP address and internal IP address. The ideal way is to use load balancer and configure it to 50:50 and change it to 100:0 when you want to do upgrade, but it costs money. A cheaper alternative is to run software load balancer on BSD vm, but it probably requires some work.
Edit: What I meant to say is session variables, not session. You said the server is stateful. If it contains some business logic that uses session variable, it needs to get stored externally to be preserved across reconnection during switch over. Actual DataSnap session will be lost, so when you shutdown web box 1 during upgrade, the client will get "Session {some-uuid} is not found" error by web box 1, and it will reconnect to web box 2.
Also you could use 3 IP addresses (1 public and 2 private) so the client always sees 1 address , which is better method.
I have done something similar by having a specific table which held my "data version". Each time I would update the server or change a system wide global setting, I would increment this field. When a client starts it always checks this value, and will check again before any transactions/queries. If the value was ever different from when I first started, then I needed to go through my re-initialization logic, which could easily include a re-login to an updated server.
I was using IIS to publish my app servers, so the data that would change would be the path to the app server. I kept the old ones available, to respond to any existing transactions that were in play. Eventually these would be removed once I knew there were no more client connections to that version.
You could easily handle knowing what versions to keep around if you log what server the client last connected too (and therefore would know about).
For newer versions (Delphi 2010 and up), there is an interesting solution
for systems using the HTTP transport:
Implementing Failover and Load Balancing in DataSnap 2010 by Andreano Lanusse
and a related question for the TCP/IP transport:
How to direct DataSnap client connections to various DS Servers?