I wonder if there is a default Robots.txt for Umbraco (7.5) websites.
Right now, I created one with these items. Shall I add anything else?
User-Agent: *
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /css
Disallow: /Errorpages/
Disallow: /scripts/
Disallow: /Config/
You should consider removing /css and /scripts from the list, actually. I believe Google finds it friendlier/better for those to be available.
Otherwise, maybe put /install/ in there as well, just as a precaution.
Regarding CSS & JSS check this: https://yoast.com/dont-block-your-css-and-js-files/. I decided to cut them off lately.
Mine, always used with Umbraco instances, robots.txt file looks like this:
# robots.txt for Umbraco
User-agent: *
Disallow: /aspnet_client/
Disallow: /bin/
Disallow: /config/
Disallow: /data/
Disallow: /install/
Disallow: /macroScripts/
Disallow: /masterpages/
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /usercontrols/
Disallow: /xslt/
User-agent: Libwww-perl
Disallow: /
Sitemap: http://{HTTP_HOST}/sitemap
I've ended up using this
Sitemap: https://{HTTP_HOST}/sitemap.aspx
User-agent: *
Disallow: /aspnet_client/
Disallow: /bin/
Disallow: /config/
Disallow: /css/
Disallow: /data/
Disallow: /install/
Disallow: /masterpages/
Disallow: /python/
Disallow: /scripts/
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /usercontrols/
Disallow: /xslt/
Related
I have a docker installation running on my machine and I can access the site on my browser by going to mysite.local (defined to map to 127.0.0.1 in hosts file)
When I run ngrok http mysite.local it works fine but when I open the http://random_id.ngrok.io link
ngrok console shows:
GET / 301 Moved Permanently
I get a NET::ERR_CERT_COMMON_NAME_INVALID error and If I click on advanced->proceed to... http://random_id.ngrok.io (unsafe), it just shows
Tunnel www.random_id.ngrok.io not found
Worth nothing this rule in .htaccess even though it looks fine to me
RewriteCond %{HTTP_HOST} !^mysite.local
RewriteRule ^(.*)$ https://www.%{HTTP_HOST}/$1 [R=301,L]
UPDATE ngrok http -host-header=rewrite mysite.local seems indeed to work but:
css and oher assets fail to load (the only css-related rule in .htaccess is: AddType text/css .chunk.css AddType text/css .css)
Looks like the rewrite rule is the issue to me.
Your rule states:
If the http host does not start with mysite.local,
then redirect to https://www.HOST/[...]
I'd recommend either removing the rewrite rule entirely or use ngrok's http host header rewrite functionality:
ngrok http -host-header=rewrite mysite.local
Jenkins is running behind IBM HTTP Webserver, and the Load Balancing is trying to monitor the Webserver for the error code to check if the Webserver is Up.
Because the security is enabled on Jenkins, GET/ call from Load Balancer is returning 403 Forbidden, is there a way this can be redirected or print "200 OK" instead, if the page is reached, rather returning "403"?
Also tried using the Rewrite engine in the httpd file
<VirtualHost *:80>
ProxyPass / http://AppServer:8080/
ProxyPassReverse / http://AppServer:8080/
ProxyRequests Off
AllowEncodedSlashes NoDecode
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^curl/*
RewriteRule ^/index.php$
</VirtualHost>
Still doesn't work. I am sure, my configuration in the httpd file not correct, any suggestions? index.php is placed in bin directory of the HTTP server.
You could use mod_rewrite to send requests for ^/$ with a user-agent of your monitoring script to some other static resource
I'm trying to configure my apache server for serving static assets from my rails app. I already tried the suggested configurations but my assets still aren't shown and when trying to access them directly I just got an rails error that no matching controller was found but the asset stuff should be handled by apache directly I think.
My apache configuration looks like this:
<VirtualHost *:80>
ServerName xxx
DocumentRoot /home/xxx/test/public
PassengerEnabled off
<LocationMatch "^/assets/.*$">
Header unset ETag
FileETag None
ExpiresActive On
ExpiresDefault "access plus 1 year"
</LocationMatch>
ProxyPass / http://127.0.0.1:9292/
ProxyPassReverse / http://127.0.0.1:9292/
</VirtualHost>
Did I missed something?
I used,
RAILS_ENV=production bundle exec rake assets:precompile
To make it all work right, I added this to config/application.rb...
module MyApp
class Application < Rails::Application
.
.
config.assets.precompile += ['custom.css']
config.assets.precompile += %w(*.png *.jpg *.jpeg *.gif)
.
.
end
end
(I had created custom.css.scss. But Rails did not recognize .scss, as you see above.) I assume that all your assets are appearing in public/assets folder after precompile. I don't understand what you are doing with LocationMatch, pardon my ignorance. Further more, I did not use port 80. I used 8000. Not sure if that makes a difference.
Also, there's a setting in config/environments/production.rb,
# Disable Rails's static asset server (Apache or nginx will already do this).
config.serve_static_assets = false
This is directly from the Rails Asset-pipeline documentation regarding the Apache server:
http://guides.rubyonrails.org/asset_pipeline.html
4.1.1 Far-future Expires Header
Precompiled assets exist on the file system and are served directly by your web server. They do not have far-future headers by default, so to get the benefit of fingerprinting you'll have to update your server configuration to add those headers.
For Apache:
# The Expires* directives requires the Apache module
# `mod_expires` to be enabled.
<Location /assets/>
# Use of ETag is discouraged when Last-Modified is present
Header unset ETag
FileETag None
# RFC says only cache for 1 year
ExpiresActive On
ExpiresDefault "access plus 1 year"
</Location>
I have a Phusion install, configured with Rails 3.0.10 and Apache 2, and assets like CSS and Javascript are apparently being served, but not utilized by the browser.
So for instance when I hit the root url, the page renders, but CSS and javascripts are not used by the browser. I see GET requests coming through for each asset in the access logs.
I can visit those assets directly in my browser, going to stylesheets/application.css?1313636333 I see the correct application.css
I dont see permission errors in production logs, or apache error logs.
My VHost looks like:
<VirtualHost *:80>
ServerAdmin info#example.com
ServerName portal.example.com
DocumentRoot /app/example/public
<Directory /app/example/public>
Options FollowSymLinks
AllowOverride None
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
Since you can access your static assets, the mime-type incorrectness might be the cause.
Take the output of the following:
curl --head http://portal.example.com/stylesheets/application.css
the result should contains "Content-Type: text/css"
I am following the threads to set up a virtual host on stackoverflow. i did follwoing
<VirtualHost *:80>
ServerAdmin timy#yt.com
DocumentRoot "C:/www/test"
ServerName test.devsite-1.com
ServerAlias test.devsite-1.com
</VirtualHost>
in windows host file
127.0.0.1 test.devsite-1.com
I hit following url in browser
http://test.devsite-1.com/
The resulting page shows me the listing of folder i.e localhost page... where as I was expecting it will show the content of index.php placed in /test folder?
Can someone correct me if I am interpreting something wrong?
Regards
If you are being shown the contents of your test folder then maybe you need to define index.php as an index.
You could attempt with a index.html file inside the directory first, but i believe you are missing the DirectoryIndex directive in your Virtualhost configuration.
Try adding:
DirectoryIndex index.html index.php
to your Virtualhost configuration.
Try adding
DirectoryIndex index.php
to your virtualhost.
BTW: I would recommend using some kind of test.devsite-1.local so you don't accidently block the real test.devsite-1.com.