Google Dataflow error on setup with no information - google-cloud-dataflow

I'm attempting to run a Beam pipeline that has a requirements.txt file. This fails and the logging entry, under worker-startup is:
{
insertId: "3570218088494260896:493966:0:74068"
jsonPayload: {
line: "boot.go:134"
message: "Failed to install packages: failed to install requirements: exit status 1"
}
labels: {
compute.googleapis.com/resource_id: "3570218088494260896"
compute.googleapis.com/resource_name: "jumps-cafc68e2-10261505-b789-harness-kn3b"
compute.googleapis.com/resource_type: "instance"
dataflow.googleapis.com/job_id: "2017-10-26_15_05_55-17840030900137737069"
dataflow.googleapis.com/job_name: "jumps-cafc68e2"
dataflow.googleapis.com/region: "global"
}
logName: "projects/sixty-capital/logs/dataflow.googleapis.com%2Fworker-startup"
receiveTimestamp: "2017-10-26T22:12:41.061522503Z"
resource: {
labels: {
job_id: "2017-10-26_15_05_55-17840030900137737069"
job_name: "jumps-cafc68e2"
project_id: "sixty-capital"
region: "global"
step_id: ""
}
type: "dataflow_step"
}
severity: "CRITICAL"
timestamp: "2017-10-26T22:12:35Z"
}
Is there a way of learning more about what happened?

Looking at the logs from the same process immediately prior show what the process was attempting to do.
In this case, the local process had created tarballs / zip files of packages from requirements.txt and uploaded them to GCS. The remote process installs requirements.txt with a link to those files - this generally allows the installation to avoid downloading anything further.
But if an item in the file is a source that the remote process can't access (either a local path, or a private git repo) rather than a simple package name, the installation fails.

Related

failed to load the 'resty.core' when loading custom APISIX plugin

I am trying to create a custom lua plugin for the APISIX docker version 2.15.0. I am using the a slightly different apisix example plugin and I am loading it using the instructions in the Developer Guide. However when I am reloading APISIX I get the following error and the plugin is not loading:
2022/10/05 14:05:40 [alert] 1#1: failed to load the 'resty.core' module (https://github.com/openresty/lua-resty-core); ensure you are using an OpenResty release from https://openresty.org/en/download.html (reason: /usr/local/apisix/apisix/plugins/3rd-party.lua:18: loop or previous error loading module 'apisix.core') in /usr/local/apisix/conf/nginx.conf:404
To reporduce:
Clone the APISIX docker repo with the docker compose stack
Create the folder <repo>/example/plugins
Create a file named 3rd-party.lua and put the code below
Edit the <repo>/apisix_conf/config.yaml and add the line extra_lua_path: "/usr/local/apisix/apisix/plugins/3rd-party.lua" under apisix
Bind the lua script to the docker by adding the line - ./plugins/3rd-party.lua:/usr/local/apisix/apisix/plugins/3rd-party.lua under the apisix's volumes section
Run the docker stack with cd ./example && docker-compose up -d
See if the plugin is loaded.
The lua plugin code:
local require = require
local core = require("apisix.core")
local plugin_name = "3rd-party"
local schema = {
type = "object",
properties = {
body = {
description = "body to replace response.",
type = "string"
},
},
required = {"body"},
}
local plugin_name = "3rd-party"
local _M = {
version = 0.1,
priority = 12,
name = plugin_name,
schema = schema,
}
function _M.check_schema(conf)
return core.schema.check(schema, conf)
end
function _M.access(conf, ctx)
return 200, conf.body
end
return _M
After taking a lot of advice from the APISIX slack the correct steps to create a plugin for the docker-APISIX version are:
Create a lua plugin script
Bind the lua script to the docker apisix service by adding the line - /path/to/plugin/script/<plugin-name>.lua:/usr/local/apisix/apisix/plugins/<plugin-name>.lua under the apisix's volumes section like:
apisix:
...
volumes:
...
- ./plugins/3rd-party.lua:/usr/local/apisix/apisix/plugins/3rd-party.lua
Get a copy of the available plugins that reside in the apisix docker container in /usr/local/apisix/conf/config-default.yaml. These are under the plugins section of the file
Add a new section named plugins in the apisix config file with the available plugins taken from the previous step like:
plugins: # plugin list (sorted by priority)
- real-ip # priority: 23000
- client-control # priority: 22000
...
Add the new plugin in the list of plugins of apisix config file like:
plugins: # plugin list (sorted by priority)
- real-ip # priority: 23000
- client-control # priority: 22000
...
- 3rd-party
The steps above will load the new plugin in APISIX and it can be validated with a call curl http://<domain>/apisix/admin/plugins/list -H 'X-API-KEY: <key>'. The new plugin should appear in the response.
The steps above will not load the plugin to the apisix dashboard beacause the Dashboard is caching the list of plugins. To reload the cache follow these instructions.

I am facing an issue while running batch command to push the code from local server to repository (Gitlab)

Here is the attached YML file and error description for more information.
Below is the yml code to execute bat file in GitLab:
job_1:
tags :
-ci
before_script:
- echo "This is the before_script."
- echo "Attempting to run the WindowsCommand 35 version-application.bat file..."
- call C:\ADM\appian-adm-versioning-client-2.5.9\version-application.bat
script:
- version-application.bat -action "addContents" -application_path "C:\Demo\Application Exports\ASD_App_12172019.zip" -commit_message "Sdlc application 12242019"
How can I make this job work?
The relevant error seems to be:
java.io.FileNotFoundException: version-manager.properties
(The system cannot find the file specified)
Check in the Appian documentation if/how that file should be present.
But you should at least version it and push it, in order for the gitlab-ci job to use it.

not able to use packages in vendor directory in docker's repository

I am trying to create a container with docker's go api. I want to expose a port using container.Config.ExposedPorts in ContainerCreate()API. Below is the code
package main
import (
"fmt"
"context"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/client"
"github.com/docker/go-connections/nat"
)
func main() {
ctx := context.Background()
cli, err := client.NewClientWithOpts(client.WithVersion("1.38"))
if err != nil {
fmt.Println("Failed to get container envoronment", err)
}
resp, err := cli.ContainerCreate(ctx, &container.Config{
Image: "hyperledger/fabric-ca",
Cmd: []string{"/bin/sh", "-c", "fabric-ca-server start -b admin:adminpw"},
Env: []string{"FABRIC_CA_HOME=/etc/hyperledger/fabric-ca-server",
"FABRIC_CA_SERVER_CA_NAME=ca.example.com"},
ExposedPorts: nat.PortSet{"22/tcp":struct{}{},},
}, nil, nil, "ca.example.com")
if err != nil {
fmt.Println(" failed to create container, err:", err)
} else {
fmt.Println(" Container ID :", resp.ID, "warning:", resp.Warnings, "err:", err)
}
}
when I compile I get the below error
vignesh#vignesh-ThinkPad-E470 ~/go-book/src/github.com/my_fabric $ go build asd.go
asd.go:8:9: cannot find package "github.com/docker/go-connections/nat" in any of:
/home/vignesh/go-book/src/github.com/my_fabric/vendor/github.com/docker/go-connections/nat (vendor tree)
/usr/local/go/src/github.com/docker/go-connections/nat (from $GOROOT)
/home/vignesh/go-book/src/github.com/docker/go-connections/nat (from $GOPATH)
As package "github.com/docker/go-connections/nat" is in a vendor directory at "github.com/docker/docker/vendor/github.com/docker/go-connections/nat", I then created a vendor directory in my working directory and copied contents of github.com/docker/docker/vendor/github.com/docker/go-connections/nat to github.com/my_fabric/vendor/go-connections/nat and used "github.com/my_fabric/go-connections/nat" in import rather than "github.com/docker/go-connections/nat". But I got the following error.
vignesh#vignesh-ThinkPad-E470 ~/go-book/src/github.com/my_fabric $ go build asd.go
# command-line-arguments
./asd.go:25:29: cannot use "github.com/my_fabric/vendor/github.com/my_fabric/go-connections/nat".PortSet literal (type "github.com/my_fabric/vendor/github.com/my_fabric/go-connections/nat".PortSet) as type "github.com/docker/docker/vendor/github.com/docker/go-connections/nat".PortSet in field value
Basically I want to use packages which is in vendor directory in docker's repository. Kindly help :)
It will work just try to provide user permissions to the vendor directory in docker environment. Beucase the vendor path to import the package is correct.
Import in golang works as:- First it will import the package from
vendor directory if not it will look for $GOPATH src directory for
the same package.
The error says it cannot find the package at any of the given paths. But you have it in the vendor directory so this can be any permission issue.
Since it happens if you are working on a linux the permissions will not allow to access the vendor directory.
Also it is better not to copy rather to generate a vendor package using Gopkg.toml in the docker.
Theses two directories are not equivalent:
github.com/docker/docker/vendor/github.com/docker/go-connections/nat
github.com/my_fabric/vendor/go-connections/nat
You have to move (not copy) all of Docker's vendored dependencies unchanged into your own vendor directory, e.g. the following directory should exist:
github.com/my_fabric/vendor/github.com/docker/go-connections/nat
Note the github.com/docker segments. If you copy the directories you'll end up with two copies of the packages, which will get you into trouble. For instance, you end up with the distinct types
"github.com/docker/docker/vendor/github.com/docker/go-connections/nat".Port
"github.com/docker/go-connections/nat".Port
You don't have to change your import statements at all.

How do I configure SaltStack to transfer a file (or install a package) for the first time?

I am running two instances of RedHat. I have SaltMaster installed on one machine and SaltMinion installed on another. I am using a free version of Salt. I want to test SaltStack to do a basic configuration management task. If it can transfer a file from SaltMaster to SaltMinion, that would be great. If it can install Apache web server on SaltMinion, that would be great. Either task will help me learn. My learning goal is semi-flexible.
I can use salt '*' test.ping. The response is True. I tried this command: salt '*' state.apply
I got this error:
> hostname.fqdn:
> Data failed to compile:
> ----------
> No matching salt environment for environment 'qa' found
> ----------
> No matching sls found for 'qa1' in env 'qa'
> ----------
> No matching sls found for 'base1' in env 'base'
> ----------
> No matching salt environment for environment 'dev' found
> ----------
> Specified SLS base1 in saltenv dev is not available on the salt master or through a configured fileserver
I modified the /etc/salt/master file. I uncommented these lines:
fileserver_backend:
- git
- roots
I tried this command again: salt '*' state.apply
I received this error:
> [ERROR ] Error parsing configuration file: /etc/salt/master -
> expected '<document start>', but found '<block mapping start>' in
> "<string>", line 547, column 1:
> fileserver_backend:
> ^ [ERROR ] Error parsing configuration file: /etc/salt/master - expected '<document start>', but found '<block mapping start>' in
> "<string>", line 547, column 1:
> fileserver_backend:
> ^
I have been following these directions here:
https://docs.saltstack.com/en/latest/topics/tutorials/states_pt1.html
I created a webserver.sls file.
I inserted these lines as the content:
apache: # ID declaration
pkg: # state declaration
- installed # function declaration
I do not see how the three lines in the directions above would be enough to configure SaltStack to work. Where would the apache installation media need to be? Where would the transfer happen from? Am I supposed to download the media to SaltMaster? I would assume so. But where would I put it? I have a satellite server for yum commands to work.
Alternatively, how do I get SaltStack to transfer a file from SaltMaster to SaltMinion?
The first error ([...]No matching sls found for 'qa1' in env 'qa'[...]) indicates that you have configured a lot of different environments (file_roots), which are not present on your master's filesystem. Your approach to solve this goes in the correct direction, but leads to this error:
[ERROR ] Error parsing configuration file: /etc/salt/master - expected '', but found '' in "", line 547, column 1: fileserver_backend: ^ [ERROR ] Error parsing configuration file: /etc/salt/master - expected '', but found '' in "", line 547, column 1: fileserver_backend: ^
You should no longer be able to test.ping your minion, as the salt master should not run anymore, does it? To solve it just read the error message. It tells you with which point in your salt master configuration file salt is unhappy.
The fileserver_backend configures which types of backend should be available. You should check the file_roots configuration to actually define which roots are available. Roots refer to salt states folders in your filesystem.
A very simple config might look like that:
file_roots:
base:
- /srv/salt
It assumes that /srv/salt is the root of your state tree - which effectively means, that your webserver.sls should be located in this folder.
Your webserver.sls looks promising - it should install apache2 on a minion, when you apply it.
Managing configuration files on the master and transferring them to the minions is something salt can easily achieve. A simple state might look like:
/etc/myawesomeconfigurationfile.conf:
file.managed:
source: salt://myawesomefile # refers to /srv/salt/myawesomefile
user: root
group: root
mode: 640
You also asked for media files that you want to manage. If you talk about application related data it is not a good idea to use salt to move them around. IMO other approaches like NFS, GlusterFS or anything else that decouples user content from your application would be a better approach.

how to deploy your dart app (using Web ui) without using Pub Deploy

What is the best strategy to deploy a Dart Web-ui app manually ?
pub deploy doesn't work for me and I have raised bug report. So am thinking what is the best way to manually deploy.
This is how I started:
1) From project root I compile the webui components (dwc.dart)
2) change directory to web/out then run dart2js
3) copy all .js files into that scripts/js public folder on server
4) copy appname.html to server changing css and script paths to option 3
5) Make sure dart.js is also in the same directory as item 3
this is as far as I got. So what else do I need to do ?
A few questions:
1) Do I manually change the file paths in the generated .js files to point to public folders on server for the files they are referencing and make sure those files are on server also ?
2) Do I need to copy all packages to server also ?
3) Any preferred file structure on server?
Any tips on this really appreciated.
Thanks.
I wrote a Grunt script for it (since I had no time to look up how to properly write code for Grunt, I did not share the code since it's a mess) but I basically do this:
compiling a list of files with dwc to a given out dir
compile it to javascript
clean up all non-deployable files
change some paths inside the HTML to match the server paths (for some reasons, this gets changed by the compilation process)
remove all packages except the ones I really need (JS interopt and browser)
Since I'm only using the JS version, I remove all dart packages. Since the paths inside the HTML files are up to you, you can already use a structure that suits you/your server.
I can provide you with a Grunt script to understand the order of tasks. Practically the order I use is this one:
Create the build directory. I usually use /build/web. I usually create these files (index.html, main.dart, /css and so on into the /web dir). I create the rest of components into /lib directory.
Compile the .dart file that contains the main() function ("main.dart" in my case for simpler projects) file to Javascript and put it into /build/web directory
Copy the other needed files and folders to the /build/web directory. Also, during this process you'll be copying the packages that your project needs. You'll see in the example provided below.
Remove all empty folders from the project
You can create a Grunt task to open the /index.html file in the browser once the building process has ended (I will not provide this example)
The structure of the dart test project:
testApp
- gruntfile.js
- package.js
/lib
/packages
/angular
/web
- index.html
- main.dart
/css
/img
So, the Grunt example script to cover steps from 1 - 4 looks like this (copy it to gruntfile.js):
module.exports = function (grunt) {
grunt.initConfig({
// 1.
// create build web directory
mkdir: {
build: {
options: {
create: ['build/web']
}
}
},
// 2.
// compile dart files
dart2js: {
options: {
// use this to fix a problem into dart2js node module. The module calls dart2js not dart2js.bat.
// this is needed for Windows. So use the path to your dart2js.bat file
"dart2js_bin": "C:/dart/dart-sdk/bin/dart2js.bat"
},
compile: {
files: {'build/web/main.dart.js': 'web/main.dart'}
}
},
// 3.
// copy all needed files, including all needed packages
// except the .dart files.
copy: {
build: {
files: [
{
expand: true,
src: [
'web/!(*.dart)',
'web/css/*.css',
'web/res/*.svg',
'web/packages/angular/**/!(*.dart)',
'web/packages/browser/**/!(*.dart)'
],
dest: 'build'
}
]
}
},
// 4.
// remove empty directories copied using the previous task
cleanempty: {
build: {
options: {
files: false
},
src: ['build/web/packages/**/*']
}
},
});
require('matchdep').filterDev('grunt-*').forEach(grunt.loadNpmTasks);
grunt.registerTask('default', [
'mkdir:build',
'dart2js',
'copy:build',
'cleanempty:build'
]);
};
So this is the Grunt script example.
Create a /gruntfile.js file into your project's root directory and copy/paste the script to it.
Create a /package.json file into your project's root directory and copy/paste the following script:
{
"name": "testApp",
"version": "0.0.1",
"description": "SomeDescriptionForTheTestApp",
"main": "",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "YourName",
"peerDependencies": {
"grunt-cli": "^0.1.13"
},
"devDependencies": {
"grunt": "^0.4.5",
"grunt-cleanempty": "^1.0.3",
"grunt-contrib-copy": "^0.7.0",
"grunt-dart2js": "0.0.5",
"grunt-mkdir": "^0.1.2",
"matchdep": "^0.3.0"
}
}
Open Command Prompt in Windows, Terminal in Linux, navigate to your project's root directory and use this command:
npm install
Wait untill all Grunt modules needed will be downloaded to your local project. Once this is finished, issue this command in Command Prompt or Terminal:
node -e "require('grunt').cli()"
You can use this to initiate Grunt default task without having Grunt installed globally on your system.
Now, to know the exact build structure for your project (including the packages that the project needs), make a build using Pub Build. Then you will be able to instruct Grunt to create the same dir structure.
You can add other tasks (like minification) if you want.
Hope this will help you all to understand the process and get you started with a test app first. Add your comments to make this even better and simplify it even more.

Resources