I'd like to scrape a javascript website using Scrapy + Splash in Google App Engine. The Splash plugin is a Docker image. Is there any way to use this within Google App Engine? App Engine itself uses a Docker image, but I'm not sure how to load and access a secondary image (which is how Splash is used). Here are the Splash install instructions
You can use Custom Runtimes in the App Engine Flexible Environment.
Custom runtimes let you build apps that run in an environment
defined by a Dockerfile. By using a Dockerfile, you can use languages
and packages that are not part of the Google Cloud Platform and use
the same resources and tooling that are used in the App Engine
flexible environment.
Explore more About Custom Runtimes. Please note when you use a custom runtime, you have to write your application code to deal with some flexible environment life-cycle and health checking requests. Check how to build a custom runtime for more information.
Deploying the Splash service separately is the proper way to accomplish this.
I went ahead and tested a few different setups and the only approach that allowed me to have Splash on App Engine was to deploy it as a custom domain, setting the forwarded_ports to able to connect directly to one of the service’s instances through its IP address.
This is clearly not an adequate solution, as it comes with many limitations and, in the end, it becomes basically using Google Compute Engine without all the control it provides.
My suggestion is that you only deploy the Scrapy service of your application to App Engine, and leave the Splash service somewhere else, like in a GCE instance.
Once you have that, all you will need to do is set a static IP address for the instance and connect to it from your App Engine app through that.
Related
I would like to enable caching in ArangoDB, automatically when my app start.
I'm using docker-compose to start the whole thing but apparently there's no simple parameter to enable caching in ArangoDB official image.
According to the doc, all the files in /docker-entrypoint-initdb.d/ are executed at container start. So I added a js file with that code:
require('#arangodb/aql/cache').properties({mode: 'on'});
It is indeed executed but caching doesn't seem to be enabled (from what I see with arangosh within the container).
My app is a JS app using arangojs, so if I can do it this way, I'd be happy too.
Thanks!
According to the performance and server config docs, you can enable caching in several ways.
Your method of adding require("#arangodb/aql/cache").properties({ mode: "on" }); to a .js file in the /docker-entrypoint-initdb.d/ directory should work, but keep an eye on the logs. You may need to redirect log output with a different driver (journals, syslog, etc.) to see what's going on. Make sure to run the command via arangosh to see if it works.
If that's a bust, you might want to see if there is a way to pass parameters at runtime (such as --query.cache-mode on). Unfortunately, I don't use Docker Compose, so I can't give you direct advice here, but try something like -e QUERY.CACHE-MODE=ON
If there isn't a way to pass params, then you could modify the config file: /etc/arangodb3/arangod.conf.
And don't forget about the REST API methods for system management. You can access AQL configuration (view and alter) in the Web UI by clicking on the Support -> Rest API -> AQL.
One thing to keep in mind - I'm not sure if the caching settings are global or tied to a specific database. View the configuration on multiple databases (including _system) to test the settings.
Google cloud to run Python program as localhost server using Cloud SDK shell
Error in google cloud shell:
ymal file codes
runtime:python37
api_version:1
threadsafe:false
handlers;
- url:/
script:test.py
test.py file code
print("hello world")
App Engine is used to host web apps.
I encourage you to have a look at Google's quickstart for App Engine and deploying a "Hello World!" app.
Your Python print("hello world") while valid Python is not a web server and won't work as-is on App Engine.
Google's documentation is comprehensive and should help you get to a running app.
Please don't include images in Stack overflow questions because these have poor accessibility.
In addition to the comments from DazWilkin,
You are using dev_appserver.py which is a development server that simulates Google App Engine (GAE) in Production. Google now discourages using that, especially for the newer runtimes like Python3. Instead Google encourages you to ..run your application in your local environment with the development tools that you usually use...
Another possible way of learning to use GAE is to look at/tinker with the code of an App which runs on GAE. For example, you can use our App, NoCommandLine, which has a hello world App in different languages including Python
a) Follow all the steps documented by Google here to install or update the Google Cloud App Engine SDK. You need to follow all the steps so that all the files are installed/copied and the necessary shortcuts created.
b) Download and install the NoCommandLine App
c) Open the App, then go to File > New Application, give your app a name (Application ID), select Python 3 under Runtime and click the 'Create' button.
d) A new application will be created. Select the App, click the 'run' icon. It will open your default browser and display 'Hello World'.
e) You can now navigate to the directory for the App and open the different files to see the structure and code and you can play around with the code
Note that to deploy your App to production i.e GAE Production, you first need to create a Project with the same name as your App running on local host and you need to provide a credit card (Google will not charge your card but you must provide it)
I want to create desktop application compatible with other OS.For that I'm using electron with angular.Because both are frame work whether it will effect performance or loading time, and also whether deploying easy,can we use all the features of angular when we use with electron like routing..?
Electron uses Chromium and NodeJS which is the reason why it is compatible with other OSs. You can talk with the NodeJS process from your angular-app which opens up some possibilities. For example opening native file-dialogs to let the user choose files. Electron also already abstracts some platform specific operations like getting the user home to save some configuration files for example.
You can use routing just like in any Angular app and I think you can use most features like you would normally but dont take me for granted on this one.
I would not say it affects your loading time to combine those too. During development you have to build your angular app before electron can start up and use those files but in production Angular is already ready to be loaded so they dont hinder each other.
I have an image with a GUI application, with base image of microsoft/windowsservercore. Application is installed correctly in the image, however I'm unable to display it on host machine. Have read several articles on this on Google and they suggest to install XServer for Windows and then we can display the application on host machine. I have been trying to run following command (as suggested in most of the articles), however it does nothing and I don't get the display. Please assist.
docker run --rm -it -e DISPLAY=127.0.0.1:0.0 eft
The DISPLAY would be useful for Linux container.
As mentioned here:
WindwosServerCore image does not come with binaries for UI applications so I doubt this will ever work in servercore image but Microsoft insiders can use new bigger WindwosServer image which I beleive have those libraries intact.
This thread adds:
I understand that you can run GUI apps but the rendered elements are not shown on any desktop. Lars Iwer [MSFT] writes in the discussion below the article:
In the container image as it is right now, GUI elements will be rendered in session 0. UI automation should work with that (e.g. programmatically searching for a window etc.).
Session 0 is the session in which all system services are run and is by definition non-interactive. Sessions, Stations and Desktops are means of isolation in Windows (NT) and whether an application can show a UI and receive user interaction depends on whether it has an access to a Station with a Desktop.
Processes in Session 0 do not have that by default.
However it used to be possible to “Allow services to interact with Desktop” and it is also possible to run interactive services in other sessions than Session 0 (pay attention to “as it is right now”). Therefore, it would be interesting to hear some expert insights from Microsoft/Docker team on that…
I've cf application which I pushed and working as expected,now I want to change some file content in RT to avoid re-push.the application in deployed to warden container so it "persist" (for this instance ) in the filesystem of the container,How can I access to this file (i've node application so I guess with the FS module) location. i.e. if I've paused app with the following structure
myApp
folder1
index.html
1.if I want to change index html content by override how should I do that?I know the path of myApp/folder1/index.html but how I know
where my app is located in the container file system?
2. There is a way when I push application to say where to put the application? I mean in the container filesystem in specific path
e.g. when you create application in windows you decide where to put it...
C:\\myApp\folder1\index.html
or
D:\\myApp\folder1\index.html
I know that maybe this is advanced question but your help is appreciated!
p.s. lets say that I've some proxy for the application in the app container which listen to the app port and this can do some changes on the files of the applications
Writing directly to the container file system is not the right approach, because Cloud Foundry containers are intended to be ephemeral and transient.
Let's say that I have one instance of an application running, in Container A, and I change the contents of folder1/index.html. If that instance fails, and is automatically restarted by Cloud Foundry, the new instance won't have the persisted changes. If I need to scale up to 3 instances of my application, then Containers B and C won't have the changed files.
Allowing Cloud Foundry to manage the container file system will assure that you have consistent, repeatable behavior in your application.
If you need to make file changes in your Cloud Foundry application instance, the two recommended approaches are:
Read and write your file from a file service that is managed by Cloud Foundry. This will ensure that all application instances are accessing the same file system, and that your changes will survive beyond the container lifecycle.
Make the changes in your application artifact, and re-push the application.