How to get the list of Expiring or dropping domains ? Just like Park.io
Thanks in advance.
First of all, you need to specify which extensions you're interested in. The possibilities will vary a lot depending on the extension.
Some registries provide so-called droplists (list of expiring domains). Others may provide zone files, from which it is possible to spot expiring domain names.
For a few extensions, the whole zone file can be obtained using the zone transfer (AXFR) mechanism supported by name servers. By comparing two versions of the zone files at different dates, you can see which names are added/removed. Removed names are usually expired/pending deletion.
Out of the few extensions listed on park.io, .to (Tonga) is the only one that currently allows AXFR.
Most of the time the registries will give you nothing. You have to compile your own lists, then check once a year if the domains get renewed or not. That can involve lots of whois scraping.
Without cooperation from the registry, or a good source, it is hard to build a complete list. In .com it's no problem. Country code domain names (ccTLDs) tend to be more protective but practice varies from one country to another.
As an example: Sweden is very open.
Expiring names in .se can be seen here:
.se-domains to be released. The zone file can be downloaded too, for data mining or other purposes: Access to zonefiles for .se and .nu
But this is the exception, not the rule.
You going to need to create your own by querying the expire WHOIS info from an API service like domaintools. An example of how to do this via URL is below:
https://www.domaintools.com/resources/api-documentation/whois-lookup/
example:
http://api.domaintools.com/v1/$myWebsite/whois/
Related
I know the first 6 characters of a domain name. I also know that the domain is about palmistry!
I need a search engine that can search all valid domain names that starts with "karath", so it should search all valid domain names on the web with search expression of "karath*"
I is better that it shoes some description about each domain.
Is there any such search engine?
If this question is not related to this community (off-topic) please tell me which community is related to my question.
You can download the complete list of all registered domains from whoisxmlapi at:
https://www.whoisxmlapi.com/whois-database-download.php
Please note that it’s a paid service, but you do not have any other option. After download the list, you can apply your desired logic (search if domain contain the string“karath”).
Keep in mind that this list is updating every day, so then you can only scan all the daily registered domains from here (free service):
https://whoisds.com/newly-registered-domains
And keep track only the delta’s.
I am working on a website. Currently the website was targeted to serve users from a specific Geographic region. Now I would like to expand its userbase to another region. The need is to serve different contents to different regions with the same base functionality.
My initial thought (I might sound a noob here) is to host the content specific to different regions on different databases -> Redirect users to specific domains and thus map the users geographically. Do suggest if its the right way to proceed.
Also, I would like to know whether there is a need to localize my website for these regions (Current language used is English)
Please post your experiences in such scenarios and also your ideas to bring about the transition.
Thanks in advance.
How do you see users being matched to their specific regional content?
Will they be presented with an option to choose?
Will you use geo functions to determine location?
Will you use server based reverse DNS lookup to determine location?
Will each region get its own "entry" URL (aka different domains)?
The first three are fraught with their own specific problems...
Presenting a choice/menu is considered bad form because it adds to the number of "clicks" necessary for a user to get to the content they actually came for.
While geo functions are very widely supported in all modern browsers, it is still seen as an issue of privacy in that a large number of users will not "allow" the functionality, meaning you'll have to fallback to a choice/menu approach anyway.
Server based reverse DNS, while a common practice, is very unreliable because many users are using VPN, proxies, TOR, etc. to specifically mask their actual location via this method of lookup.
Personally, my experience is to use completely separate entry URLs that are all hosted as virtual domains on a single Web Server. This gives you a large array of methods of determining which entry URL was used to access your code, and then format/customize the content appropriately.
There is really no need to setup separate servers and/or databases to handle these different domains/regions.
With that said, even if the language is common across regions, it is a very good habit to configure your servers and databases to support UTF-8 end-to-end, such that if any language specific options need to be supported in the future, then you won't need to change your code to do so. This is especially true if your site will capture any user generated input.
Is there an ATOM Client or framework that enables capture of a feed entry EXACTLY once?
If not, what is the best architecture?
We would like to parse an ATOM feed, persisting all syndication feed entries locally as individual records (in database or file system). These records may have to be deleted periodically for efficiency. So client must keep track of which entries it has already looked at, independently - of the said persistence.
Have you looked at Superfeedr? Its a software as a Service platform that does just that: fetches feeds, parsed them and sends the new entries to your endpoints when they're available.
Answering my own, per working solution developed. In one word, the architectural solution to capturing only new and unique entries from a syndication feed is - CACHING.
Specifically, the entries must be stored by the client to support the logic "does the feed have anything new for me?". I believe there is no shortcut to this "client-side" solution.
Conditional-GET is not a complete solution, even if supported server side by the syndicated feed. For instance, if the client client does not send the exact If-Modified-Since time-stamp, the server can ignore the header and simply generate all entries again. Per Chris Berry, Bryon Jacob. Updated 10/27/08,
...a better way to do this is to use the start-index parameter, where the client sets this value to the end-Index, returned in the previous page. Using start-index ensures that the client will never see the same response twice, or miss Entries, since several Entries could have the same "update date", but will never have the same "update index".
In short, there is no standard server side solution guaranteeing "new/unique". The very idea of "uniqueness" is a client-side issue anyway and same opinion may not be shared by the server. From that perspective, it would be impossible for server to satisfy all clients. And, anyway, the question does not pertain to developing a better syndication server, but a smarter client, therefore, Caching is the way to go.
Cache implementation must persist between feed polls and the time-to-live (ttl), time-to-idle (tti) properties of entries stored in cache must be set appropriately to BOTH limit the performance/size of cache AND also to adequately cover the feed's oldest entries between polling cycles. Cache could be memory resident, database, file system or network array. A product like EHCache, ehcache.org, provides just about all the functionality needed.
The feed entries may persisted as-is, but the best (most efficient) method would be to identify contents, or combinations thereof, that make them unique. Methods like serialization in java or Google's Protocol Buffers may be used to create unique compact keys to persist in the cache. In my simple solution, I did not even bother storing the entries, just the keys generated as an MD5 hash of a couple of entry fields by which I defined how an entry would be unique for my purpose.
Hope this flowchart is helpful.
I believe this question is platform/technology independent, however I am using Ruby on Rails with the carrierwave gem.
Users upload documents to my site, and I need to keep them private. I am exploring the different options available to me, along with their advantages and disadvantages.
Option 1
Obfuscate urls to images to make them impossible to guess.
This would be relatively simple to implement and fast to serve up. However, if a url was made public by whatever means, security is lost.
Option 2
Have documents accessed through some sort of intermediate step that requires authentication. This would have improved security over option 1, but would place additional load on the server. A page containing previews of a number of uploaded documents would hammer the server.
Are there any other options available to me? Have I made any mistakes with my claims, or missed any important points?
I think the best option you have is to have a "key" for your documents. You can generate a key, with a certain lifetime, and when you go on /document/name/access_key, you find the record matching and return the file associated with the record. Never exposing the real URL.
I'm working on a service to provide our students and faculty with one single calendar (ICS subscription) of their academic dates (start and end of term & exam periods, class dates and times, exam dates and times, etc). I'm a Ruby and Rails newbie; we're starting to work more with it here so I figured this would be a good learning opportunity. I'm having trouble figuring out how to structure and model (if at all) certain parts of the app.
The app is conceptually pretty basic:
User logs in and a user record is created for them. A UUID is generated and stored on the user's record; it's used to generate their ICS URL (http://myservice.foo/feeds/johndoe_ce4970706f320130588b109add5c7cb0.ics).
When the user requests their ICS file (through the above URL), I need to query a bunch of different systems in order to get information in order to build a calendar:
The Student Information System (SIS) contains the user's schedule (e.g. johndoe is taking ENGL 100 on MWF from 10:30 - 11:20). I need to parse this data and create events.
Our online learning management system, Canvas, provides a calendar of assignments for courses contained inside it. It's accessible as an ICS file, so I need to pull down that file, parse it and include it in the "master" calendar that my app will generate.
Instructors can specify an additional ICS URL for their course so that they can include arbitrary events not provided by either of the two above sources. Like the Canvas calendar, I need to download and parse that ICS.
I have the first part working; I can log in through our single sign-on system (CAS) and a user record is created with a generated UUID. I'm not sure how to handle the second part, though. I don't need to store much permanent data; basically, I only need to keep around the user record (which contains their username, the generated UUID, and some access tokens for the Canvas LMS). The feed will be generated the first time it's requested, cached for some period (and regenerated on-demand when needed).
Where should I be putting the parsing and generating code? I'd like it to be somewhat modular as I expect that we'd be adding other data sources as they crop up. Should I be creating calendar and event models if I'm not actually persisting that data?
No, there is no need to create an empty model for interaction with 3rd party services. I had a similar problem, where I needed to receive data from an external service and wanted it to be modular. One of recommended solutions I found was to create a class (that handles business logic of the interaction with the external service) in the "lib" folder under the root directory of your rails project.
It later can be required in your controller and used to receive date from the third party service. Or if you want it autoloaded, then you can add path to lib directory in your application.rb file under config.autoload_paths setting.