I followed this tutorial and leveraged this github project and wrote a simple 'mysum' UDF function.
#ScalarFunction("mysum")
#Description("Returns summation of two numbers")
#SqlType(StandardTypes.BIGINT)
public static long sum(#SqlType(StandardTypes.BIGINT) long num1, #SqlType(StandardTypes.BIGINT) long num2)
{
return num1 + num2;
}
I followed the below steps to plugin the UDF but Presto fails to import the function. Since, there is very little documentation about writing and plugging UDFs, any help would be much appreciated. Thanks in advance.
ran mvn compile and mvn package
copied the .jar file into plugins folder under presto (unzipped) directory /Users/nithin/presto-server-0.166/plugin/udfs/
started coordinator using bin/launcher run
ran select mysum(10,100) in Presto CLI, but throws error
Below is error log when I try to run my UDF ('mysum') in Presto CLI. It is quite evident that Presto is not able to find the UDF, so plugging in wasn't successful. How to fix that? Am I missing any step?
➜ Workspaces ./presto.jar --server localhost:8080 --catalog mysql --schema default --debug
presto:default> select mysum(99,100);
Query 20170228_183509_00002_vr5dt failed: line 1:8: Function mysum not registered
com.facebook.presto.sql.analyzer.SemanticException: line 1:8: Function mysum not registered
at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitFunctionCall(ExpressionAnalyzer.java:824)
at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.visitFunctionCall(ExpressionAnalyzer.java:255)
at com.facebook.presto.sql.tree.FunctionCall.accept(FunctionCall.java:111)
at com.facebook.presto.sql.tree.StackableAstVisitor.process(StackableAstVisitor.java:26)
at com.facebook.presto.sql.analyzer.ExpressionAnalyzer$Visitor.process(ExpressionAnalyzer.java:274)
at com.facebook.presto.sql.analyzer.ExpressionAnalyzer.analyze(ExpressionAnalyzer.java:231)
at com.facebook.presto.sql.analyzer.ExpressionAnalyzer.analyzeExpression(ExpressionAnalyzer.java:1406)
at com.facebook.presto.sql.analyzer.StatementAnalyzer.analyzeExpression(StatementAnalyzer.java:1802)
at com.facebook.presto.sql.analyzer.StatementAnalyzer.analyzeSelect(StatementAnalyzer.java:1623)
at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitQuerySpecification(StatementAnalyzer.java:800)
at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitQuerySpecification(StatementAnalyzer.java:188)
at com.facebook.presto.sql.tree.QuerySpecification.accept(QuerySpecification.java:127)
at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitQuery(StatementAnalyzer.java:550)
at com.facebook.presto.sql.analyzer.StatementAnalyzer.visitQuery(StatementAnalyzer.java:188)
at com.facebook.presto.sql.tree.Query.accept(Query.java:94)
at com.facebook.presto.sql.tree.AstVisitor.process(AstVisitor.java:27)
at com.facebook.presto.sql.analyzer.Analyzer.analyze(Analyzer.java:68)
at com.facebook.presto.sql.analyzer.Analyzer.analyze(Analyzer.java:60)
at com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:290)
at com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:276)
at com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:234)
at com.facebook.presto.execution.QueuedExecution.lambda$start$1(QueuedExecution.java:63)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
select mysum(99,100)
Your connector plugin should provide the list of UDF classes. See Plugin#getFunctions
default Set<Class<?>> getFunctions()
{
return emptySet();
}
Please override the method and expose classes of UDF classes.
What I usually prefer is to build the presto UDF jar locally and try to install on dockerized presto cluster to make sure, it gets installed successfully.
I have used gradle to build my jar and used the latest modules after presto foundation launch as dependencies. While in maven, the file in meta-inf/services is automatically created, In gradle i had to set it up manually.
Here is the github repo and steps to test the UDF locally.
Related
Getting the following error on neo4j server startup with the apoc plugin.
Have copied the jar to the plugins folder.
Caused by:
org.neo4j.kernel.api.exceptions.ComponentInjectionException: Unable to
set up injection for procedure 'Static', the field 'apocConfig' has
type 'class apoc.ApocConfig' which is not a known injectable component
Neo4j version: 4.0.4
Apoc version: 4.0.0.13
What could be the problem?
I would suggest reviewing the installation procedure to see if you missed any steps,
https://neo4j.com/docs/operations-manual/current/installation/linux/
Also, double check the java version that neo4j is using, and check if there are any other local java environment factors specific to this install. class paths, other jars, etc.
We were having the same issue with neo4j 4.x and custom plugin folder.
Updating custom plugin folder in neo4j config didn't work, we had to add folder (or extension) into java CP (classpath) as well.
It worked in 3.x neo4j version without adding a folder into the classpath.
Try to make sure that you have plugins folder listed in -cp and might work for you.
Similar to Ilija's problem, our embedded database (using Eclipse with the Maven plugin) did not like our custom plugin folder.
Since the database was for setup and then deployment (effectively 2 separate things), we could move the database from the default Neo4j directory after it was setup.
As a test, you could delete the contents of your plugin folder and see if it works then.
I'm in the process of migrating entire CloudFormation stacks to Troposphere, including Lambda and Lambda-reliant CFN Custom Resources.
One of my goals is to circumvent the creation of template files altogether, making the Python code the sole "source of truth" (i.e without template files that are created and therefore can be edited, causing config drift).
This requires the ability to:
Pass a file-like object to the SAM builder (instead of a file-name)
Calling the AWS SAM builder from Python and not the CLI
My first naive idea was that I would be able to import a few modules from aws-sam-cli put a wrapper for io.StringIO around it (to hold the template as file-like object) and presto! Then I looked at the source code for sam build and all hope left me:
I may not be able to use Docker/containers for building, as I it will map the build environment, including template files.
AWS SAM CLI is not designed to have a purely callable set of library functions, similar to boto3. Close, but not quite.
Here is the core of the Python source
with BuildContext(template,
base_dir,
build_dir,
clean=clean,
manifest_path=manifest_path,
use_container=use_container,
parameter_overrides=parameter_overrides,
docker_network=docker_network,
skip_pull_image=skip_pull_image,
mode=mode) as ctx:
builder = ApplicationBuilder(ctx.function_provider,
ctx.build_dir,
ctx.base_dir,
manifest_path_override=ctx.manifest_path_override,
container_manager=ctx.container_manager,
mode=ctx.mode
)
try:
artifacts = builder.build()
modified_template = builder.update_template(ctx.template_dict,
ctx.original_template_path,
artifacts)
move_template(ctx.original_template_path,
ctx.output_template_path,
modified_template)
click.secho("\nBuild Succeeded", fg="green")
msg = gen_success_msg(os.path.relpath(ctx.build_dir),
os.path.relpath(ctx.output_template_path),
os.path.abspath(ctx.build_dir) == os.path.abspath(DEFAULT_BUILD_DIR))
click.secho(msg, fg="yellow")
This relies on a number of imports from a aws-sam-cli internal library with the build focused ones being
from samcli.commands.build.build_context import BuildContext
from samcli.lib.build.app_builder import ApplicationBuilder, BuildError, UnsupportedBuilderLibraryVersionError, ContainerBuildNotSupported
from samcli.lib.build.workflow_config import UnsupportedRuntimeException
It's clear that this means it's not as simple as creating something like a boto3 client and away I go! It looks more like I'd have to fork the whole thing and throw out nearly everything to be left with the build command, context and environment.
Interestingly enough, sam package and sam deploy, according to the docs, are merely aliases for aws cloudformation package and aws cloudformation deploy, meaning those can be used in boto3!
Has somebody possibly already solved this issue? I've googled and searched here, but haven't found anything.
I use PyCharm and the AWS Toolkit which if great for development and debugging and from there I can run SAM builds, but it's "hidden" in the PyCharm plugins - which are written in Kotlin!
My current work-around is to create the CFN templates as temp files and pass them to the CLI commands which are called from Python - an approach I've always disliked.
I may put in a feature request with the aws-sam-cli team and see what they say, unless one of them reads this.
I've managed to launch sam local start-api from a python3 script.
Firstly, pip3 install aws-sam-cli
Then the individual command can be imported and run.
import sys
from samcli.commands.local.start_api.cli import cli
sys.exit(cli())
... provided there's a template.yaml in the current directory.
What I haven't (yet) managed to do is influence the command-line arguments that cli() would receive, so that I could tell it which -t template to use.
Edit
Looking at the way aws-sam-cli integration tests work it seems that they actually kick off a process to run the CLI. So they don't actually pass a parameter to the cli() call at all :-(
For example:
class TestSamPython36HelloWorldIntegration(InvokeIntegBase):
template = Path("template.yml")
def test_invoke_returncode_is_zero(self):
command_list = self.get_command_list(
"HelloWorldServerlessFunction", template_path=self.template_path, event_path=self.event_path
)
process = Popen(command_list, stdout=PIPE)
return_code = process.wait()
self.assertEquals(return_code, 0)
.... etc
from https://github.com/awslabs/aws-sam-cli/blob/a83aa9e620ff679ca740496a3f1ff4872b88894a/tests/integration/local/invoke/test_integrations_cli.py
See also start_api_integ_base.py in the same repo.
I think on the whole this is to be expected because the whole thing is implemented in terms of the click command-line application framework. Unfortunately.
See for example http://click.palletsprojects.com/en/7.x/testing/ which says "The CliRunner.invoke() method runs the command line script in isolation ..." -- my emphasis.
I am using following python script to run sam cli commands. This should work for you too.
import json
import sys
import os
try:
LAMBDA_S3_BUCKET="s3-bucket-name-in-same-region"
AWS_REGION="us-east-1"
API_NAME = "YourAPIName"
BASE_PATH="/path/to/your/project/code/dir"
STACK_NAME="YourCloudFormationStackName"
BUILD_DIR="%s/%s" % (BASE_PATH, "build_artifact")
if not os.path.exists(BUILD_DIR):
os.mkdir(BUILD_DIR)
os.system("cd %s && sam build --template template.yaml --build-dir %s" % (BASE_PATH, BUILD_DIR))
os.system("cd %s && sam package --template-file %s/template.yaml --output-template-file packaged.yaml --s3-bucket %s" %(BASE_PATH, BUILD_DIR, LAMBDA_S3_BUCKET))
os.system("cd %s && sam deploy --template-file packaged.yaml --stack-name %s --capabilities CAPABILITY_IAM --region %s" %(BASE_PATH, STACK_NAME, AWS_REGION))
except Exception as e:
print(e.message)
exit(1)
I am trying to migrate a huge project having visual studio and maven projects to bazel. I need to access our in house maven server which is encrypted. To get access I need the load the maven_jar skylark extension since the default impl does not support encryption (get error 401). using the extension leads to a lot of troubles, like:
ERROR: BUILD:4:1: no such package '#org_bouncycastle_bcpkix_jdk15on//jar': Traceback (most recent call last):
File ".../external/bazel_tools/tools/build_defs/repo/maven_rules.bzl", line 280
_maven_artifact_impl(ctx, "jar", _maven_jar_build_file_te...)
File ".../external/bazel_tools/tools/build_defs/repo/maven_rules.bzl", line 248, in _maven_artifact_impl
fail(("%s: Failed to create dirs in e...))
org_bouncycastle_bcpkix_jdk15on: Failed to create dirs in execution root.
The main issue seems to be the shell that needs to be provided to bazel in BAZEL_SH environment variables:
I am working under windows
I am using bazel 0.23.2
bazel seems to run a bash command using "bash" directly and not the one provided by env variable.
I got a ubuntu shell installed in windows. bazel was using everything from ubuntu, especially when using maven (settings.xml was using from ubuntu ~/.m2 and not from windows user)
after uninstalling ubuntu and making sure that bash in a cmd ends up in "command not found" I also removed the BAZEL_SH env var and bazel throws the message above
after setting the BAZEL_SH variable again it fails with the same error message
I am assuming that bazel gets a bash from somewhere or is ignoring the env variable. My questions are:
1. How to setup a correct shell?
2. Is BAZEL_SH needed when using current version?
For me the doc at bazel website about setup is outdated.
Cheers
Please consider using rules_jvm_external to manage your Maven dependencies. It supports both Windows and private repositories using HTTP Basic Authentication.
For me the doc at bazel website about setup is outdated.
The Bazel team is aware of this and will be updating our docs shortly.
I have an jhipster-spring boot application which is running docker. I want to create war file. I found some commands here:
https://jhipster.github.io/production/
When I type ./mvnw -Pdev package .I am typing this command in intellij terminal. I am getting this error: http://paste.ubuntu.com/23082073/
I have no idea what is it because normally my app is running correcty. What I need for creating war? Should I connect database? There is no more info in above link.
Thanks
from the logs that you pasted, it looks an error from spring and hibernate. check if the connections to database are working, check if your database is setup correctly as in the logs found this
Wrong column type in FRAUD.PUBLIC.MODEL_WEIGHT for column weight. Found: double, expected: real
after that use the command ./mvnw -Pprod package to generate the WAR file.
Hope this helps.
I created plugin in neo4j by next steps:
1) create *.class from *.java (copile with Eclipse)
2) put *.class into .../org/neo4j/server/plugins/
3) create *.jar (using jar)
4) put into *.jar/META-INF/services/ file "org.neo4j.server.plugins.ServerPlugin" with text "org.neo4j.server.plugins.TransportRouter".
5) put *.jar into .../neo4j/plugins/
6) restart server
But my plugin do not see in "extensions" ("curl localhost:7474/db/data/").
Why?
TIA, Eugeny.
Adding this answer for the benefit of others. I had similar problem but it was not a issue with the code. You need to make sure neo4j server is stopped before you copy the jar in plugins directory. If you copy it before its stopped then restart it. It will not work. I tried this on both 1.8.2 and 1.9.2
Simple steps to be added for installing plugin
Stop the neo4j server
Copy the plugin jar file in plugins directory.
start the neo4j server
Your plugin should be detected. This issue was reproducible with the example plugin GetAll as well. Hope this helps others.
Eugeny can you list the content of the jar jar tf your-jar.jar and the source code of your class.
Do you depend on any other libraries?
please see the Neo4j manual for the needed steps.
Problem was solved.
It's my error in initialization of my class:
public TransportRouter( EmbeddedReadOnlyGraphDatabase graphdb ) {
It was replaced by:
public TransportRouter() {