Header parameters describing an API with Swagger - swagger

I'm trying to use Swagger to create the spec for a Connexion API (Python+Flask). Great tools. I know that HTTP Request Headers** are not passed to the handler functions as regular parameters but I need to be able to get request headers from the operations. I read https://connexion.readthedocs.io/en/latest/request.html#header-parameters . I used the Swagger Editor to generate a minimal python server (proof of concept) but it doesn't work from the scratch, it may be a problem in requirements:
The default requirements.txt doesn't allow me to launch the server, showing this error message:
$ python3 -m swagger_server
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/agalindodev/tmp/python-flask-server/swagger_server/__main__.py", line 3, in <module>
import connexion
File "/home/agalindodev/tmp/python-flask-server/venv/lib/python3.6/site-packages/connexion/__init__.py", line 3, in <module>
from .apis import AbstractAPI # NOQA
File "/home/agalindodev/tmp/python-flask-server/venv/lib/python3.6/site-packages/connexion/apis/__init__.py", line 1, in <module>
from .abstract import AbstractAPI # NOQA
File "/home/agalindodev/tmp/python-flask-server/venv/lib/python3.6/site-packages/connexion/apis/abstract.py", line 14, in <module>
from ..operation import Operation
File "/home/agalindodev/tmp/python-flask-server/venv/lib/python3.6/site-packages/connexion/operation.py", line 7, in <module>
from .decorators import validation
File "/home/agalindodev/tmp/python-flask-server/venv/lib/python3.6/site-packages/connexion/decorators/validation.py", line 9, in <module>
from werkzeug import FileStorage
ImportError: cannot import name 'FileStorage'
and modifing requirements.txt moving from connexion==1.1.15 to connexion==2.6.0 it launches but I finaly get:
TypeError: my_job_create() missing 1 required positional argument: 'my_session'
This is my environment:
1. OS and runtime:
python 3.6.9 on Ubuntu 18.04
2. requirements.txt
# connexion == 1.1.15
connexion == 2.4.0
python_dateutil == 2.6.0
setuptools >= 21.0.0
3. Complete swagger spec:
swagger: "2.0"
basePath: /api
info:
title: "Just a swagger test API"
version: "1.0.0"
paths:
/my_jobs:
post:
operationId: my_job.create
tags:
- MyJob
summary: "Create a job"
consumes:
- "application/json"
produces:
- "application/json"
parameters:
- name: "my_session"
in: "header"
description: "Session id that's creating the job"
required: True
type: string
responses:
"201":
description: "Successfully created a job"
schema:
$ref: "#/definitions/MyJob"
definitions:
MyJob:
type: "object"
properties:
id:
type: "string"
4. the error:
Using the modified requirements.txt I simply tried to POST a creation, passing the header but it generates an error:
$ curl -v -X POST --header 'Content-Type: application/json' --header 'Accept: application/problem+json' --header 'my_session: { "id": "xxxxx" }' 'http://0.0.0.0:8080/api/my_jobs'
* Trying 0.0.0.0...
* TCP_NODELAY set
* Connected to 0.0.0.0 (127.0.0.1) port 8080 (#0)
> POST /api/my_jobs HTTP/1.1
> Host: 0.0.0.0:8080
> User-Agent: curl/7.58.0
> Content-Type: application/json
> Accept: application/problem+json
> my_session: { "id": "xxxxx" }
>
* HTTP 1.0, assume close after body
< HTTP/1.0 500 INTERNAL SERVER ERROR
< Content-Type: application/problem+json
< Content-Length: 252
< Server: Werkzeug/0.12.2 Python/3.6.9
< Date: Sun, 02 Aug 2020 10:39:58 GMT
<
{
"detail": "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.",
"status": 500,
"title": "Internal Server Error",
"type": "about:blank"
}
* Closing connection 0
the generated swagger server dumps this output:
[2020-07-31 14:08:50,078] ERROR in app: Exception on /api/my_jobs [POST]
Traceback (most recent call last):
File "/.../python-flask-server/venv/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/.../python-flask-server/venv/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/.../python-flask-server/venv/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/.../python-flask-server/venv/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/.../python-flask-server/venv/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/.../python-flask-server/venv/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/.../python-flask-server/venv/lib/python3.6/site-packages/connexion/decorators/decorator.py", line 48, in wrapper
response = function(request)
File "/.../python-flask-server/venv/lib/python3.6/site-packages/connexion/decorators/uri_parsing.py", line 173, in wrapper
response = function(request)
File "/.../python-flask-server/venv/lib/python3.6/site-packages/connexion/decorators/validation.py", line 388, in wrapper
return function(request)
File "/.../python-flask-server/venv/lib/python3.6/site-packages/connexion/decorators/parameter.py", line 126, in wrapper
return function(**kwargs)
TypeError: my_job_create() missing 1 required positional argument: 'my_session'
127.0.0.1 - - [31/Jul/2020 14:08:50] "POST /api/my_jobs HTTP/1.1" 500 -
How can I make it work?
Many thanks!!!

Header parameters are not passed to the handler functions as parameters
https://connexion.readthedocs.io/en/latest/request.html#header-parameters.
You cant use
parameters:
- name: "my_session"
in: "header"
description: "Session id that's creating the job"
required: True
type: string
You must use connexion.request.headers['my_session']

Related

Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))

my code is ok but it is returning internal server error
below is the error code
C:\Users\Aisha\anaconda3\python.exe "C:/Program Files/JetBrains/PyCharm Community Edition 2022.3.2/plugins/python-ce/helpers/pydev/pydevd.py" --multiprocess --qt-support=auto --client 127.0.0.1 --port 52852 --file C:\Users\Aisha\Codesample\KAFDISEASEDETECTION\API\tfservingversion.py
Connected to pydev debugger (build 223.8617.48)
INFO: Started server process [15480]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
INFO: ::1:52885 - "POST /predict HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
raise err
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 1042, in _validate_conn
conn.connect()
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x0000029197A48D00>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Aisha\anaconda3\lib\site-packages\requests\adapters.py", line 489, in send
resp = conn.urlopen(
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "C:\Users\Aisha\anaconda3\lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='localhost', port=8501): Max retries exceeded with url: /v1/models/MyModels:predict (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000029197A48D00>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Aisha\anaconda3\lib\site-packages\uvicorn\protocols\http\httptools_impl.py", line 419, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "C:\Users\Aisha\anaconda3\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in call
return await self.app(scope, receive, send)
File "C:\Users\Aisha\anaconda3\lib\site-packages\fastapi\applications.py", line 270, in call
await super().call(scope, receive, send)
File "C:\Users\Aisha\anaconda3\lib\site-packages\starlette\applications.py", line 124, in call
await self.middleware_stack(scope, receive, send)
File "C:\Users\Aisha\anaconda3\lib\site-packages\starlette\middleware\errors.py", line 184, in call
raise exc
File "C:\Users\Aisha\anaconda3\lib\site-packages\starlette\middleware\errors.py", line 162, in call
await self.app(scope, receive, _send)
File "C:\Users\Aisha\anaconda3\lib\site-packages\starlette\middleware\cors.py", line 84, in call
await self.app(scope, receive, send)
File "C:\Users\Aisha\anaconda3\lib\site-packages\starlette\middleware\exceptions.py", line 79, in call
raise exc
File "C:\Users\Aisha\anaconda3\lib\site-packages\starlette\middleware\exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "C:\Users\Aisha\anaconda3\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in call
raise e
File "C:\Users\Aisha\anaconda3\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "C:\Users\Aisha\anaconda3\lib\site-packages\starlette\routing.py", line 706, in call
await route.handle(scope, receive, send)
File "C:\Users\Aisha\anaconda3\lib\site-packages\starlette\routing.py", line 276, in handle
await self.app(scope, receive, send)
File "C:\Users\Aisha\anaconda3\lib\site-packages\starlette\routing.py", line 66, in app
response = await func(request)
File "C:\Users\Aisha\anaconda3\lib\site-packages\fastapi\routing.py", line 235, in app
raw_response = await run_endpoint_function(
File "C:\Users\Aisha\anaconda3\lib\site-packages\fastapi\routing.py", line 161, in run_endpoint_function
return await dependant.call(**values)
File "C:\Users\Aisha\Codesample\KAFDISEASEDETECTION\API\tfservingversion.py", line 68, in predict
response = requests.post(endpoint, json=json_data, verify=False, timeout=5)
File "C:\Users\Aisha\anaconda3\lib\site-packages\requests\api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
File "C:\Users\Aisha\anaconda3\lib\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\Aisha\anaconda3\lib\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\Aisha\anaconda3\lib\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "C:\Users\Aisha\anaconda3\lib\site-packages\requests\adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost', port=8501): Max retries exceeded with url: /v1/models/MyModels:predict (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000029197A48D00>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
Process finished with exit code -1
from wsgiref import headers
from fastapi import FastAPI, File, UploadFile
import uvicorn
import numpy as np
from io import BytesIO
from PIL import Image
import tensorflow as tf
import requests
import json
import ssl
# import urllib3
from fastapi.middleware.cors import CORSMiddleware
# Create an SSL context with the desired options
ssl_context = ssl.create_default_context()
ssl_context.options |= ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1
# urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
app = FastAPI()
origins = [
"http://localhost",
]
app.add_middleware(
CORSMiddleware,
allow_origins=origins,
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
# '' # ''beta_model = tf.keras.model.load_model("../MyModels/2")
endpoint = "https://localhost:8501/v1/models/MyModels:predict"
CLASS_NAMES = ["Banana_cordana", "Banana_healthy", "Banana_pestalotiopsis", "Banana_sigatoka",
"Corn_Cercospora_Grayleaf_spot", "Corn_Common_rust", 'Corn_Northern_Leaf_Blight', 'Corn_healthy',
'Potato___Early_blight', 'Potato___Late_blight', 'Potato___healthy', 'Rice_Bacterial _Leaf_blight',
'Rice_Brown_Spot', 'Rice_Leaf_ Smut', 'Tomato_Bacterial_spot', 'Tomato_Early_blight',
'Tomato_Late_blight', "Tomato_Leaf_Mold", "Tomato_Septoria_leaf_spot",
"Tomato_Spider_mites_Two_spotted_spider_mite", "Tomato__Target_Spot",
"Tomato__Tomato_YellowLeaf__Curl_Virus", "Tomato__Tomato_mosaic_virus", "Tomato_healthy"]
#app.get("/ping")
async def ping():
return " How are you today "
def read_file_as_image(data) -> np.ndarray:
image = np.array(Image.open(BytesIO(data)))
return image
#app.post("/predict")
async def predict(
file: UploadFile = File(...)
):
image = read_file_as_image(await file.read())
img_batch = np.expand_dims(image, 0)
json_data = {
"instances": img_batch.tolist()
}
response = requests.post(endpoint, json=json_data, verify=False, timeout=5)
# Make a request to the TensorFlow Serving endpoint with the SSL context
# response = requests.post('https://localhost:8501/v1/models/your-model:predict', json=json_data,
# verify=False, timeout=5, cert=None, headers=headers, auth=None, proxies=None, stream=None, allow_redirects=True, proxies_auth=None, **ssl_context)
# predictions = json.loads(response.text)['predictions'][0]
# predicted_class_index = np.argmax(predictions)
# predicted_class = CLASS_NAMES[predicted_class_index]
# confidence = str(round(100 * (predictions[predicted_class_index]), 2)) + "%"
# return {"class": predicted_class, "confidence": confidence}
pass`your text`
if __name__ == "__main__":
uvicorn.run(app, host='localhost', port=8000)
this is my code
what i expected response 200 ok i tried to change the port number , update python, download openssl,update tensorflow and other dependencies
instead i got internal server error from my postman and the following error from pycharm
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='localhost', port=8501): Max retries exceeded with url: /v1/models/MyModels:predict (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x0000029197A48D00>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))

Local CosmosDB connectivity with pymongo for gramex

Not able to connect cosmos db using gramex mongo adapter.
From python, We can connect to local cosmosDB instance using:
# --------------------------------------
import pymongo
uri = r"mongodb://localhost:Tm%2BOzpb8BrV7DrZHSrGm3GMKyx9r%2Frl5ue9letmD1XRUUiafHFUyIQNenAQDla85nqVDrb8tr%2FtB0LR4azi1FQ%3D%3D#localhost:10255/admin?ssl=true"
client = pymongo.MongoClient(uri,
tls=True,
tlsCAFile='./documentdbemulatorcert.cer')
db = client.admin
print(db.command("serverStatus"))
# --------------------------------------
Please mind “tlsCAFile” parameter
However in gramex when I connect using:
# --------------------------------------
url:
envvartest-app-data:
pattern: /$YAMLURL/appdata
handler: FormHandler
kwargs:
url: "mongodb://localhost:Tm%2BOzpb8BrV7DrZHSrGm3GMKyx9r%2Frl5ue9letmD1XRUUiafHFUyIQNenAQDla85nqVDrb8tr%2FtB0LR4azi1FQ%3D%3D#localhost:10255/admin?ssl=true"
database: galaxy-dev
collection: threats
id: record_number
connect_args:
tls: True
tlsCAFile: $YAMLPATH/documentdbemulatorcert.cer
# ssl:
# ssl_ca: $YAMLPATH/documentdbemulatorcert.cer
# --------------------------------------
the connection fails reading:
ERROR 17-Feb 14:05:06 formhandler 9988 envvartest-app-data: filter failed
Traceback (most recent call last):
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\handlers\formhandler.py", line 158, in get
result[key] = yield val
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\tornado\gen.py", line 1133, in run
value = future.result()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\_base.py", line 428, in result
return self.__get_result()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\data.py", line 235, in filter
data = method(url=url, controls=controls, args=args, query=query, **kwargs)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\data.py", line 1474, in _filter_mongodb
meta_cols = pd.DataFrame(list(table.find().limit(100)))
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\cursor.py", line 1238, in next
if len(self.__data) or self._refresh():
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\cursor.py", line 1130, in _refresh
self.__session = self.__collection.database.client._ensure_session()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\mongo_client.py", line 1935, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\mongo_client.py", line 1883, in __start_session
server_session = self._get_server_session()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\mongo_client.py", line 1921, in _get_server_session
return self._topology.get_server_session()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\topology.py", line 520, in get_server_session
session_timeout = self._check_session_support()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\topology.py", line 502, in _check_session_support
None)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\topology.py", line 220, in _select_servers_loop
(self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: localhost:10255: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1091), Timeout: 30s, Topology Description: <TopologyDescription id: 620e089c395cf9273486aa57, topology_type: Single, servers: [<ServerDescription ('localhost', 10255) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:10255: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1091)')>]>
ERROR 17-Feb 14:05:06 __init__ 9988 500 GET /appdata (127.0.0.1) 30795.58ms envvartest-app-data
Not able to connect cosmos db using gramex mongo adapter.
Steps to reproduce
Install CosmosDB emulator
Generate access key:
.\Microsoft.Azure.Cosmos.Emulator.exe /GenKeyFile=D:\99exps\cosmosdb\key
Start cosmosDB emulator with mongoDB support:
.\Microsoft.Azure.Cosmos.Emulator.exe /FailOnSslCertificateNameMismatch /EnableMongoDbEndpoint=3.6 /EnableMongoDbEndpoint=3.2 /KeyFile=D:\99exps\cosmosdb\key
Create Gramex Application with above mentioned configuration
Observe the error
Edit 1:
The error is changed now:
ERROR 19-Feb 00:10:33 formhandler 9988 galaxy-app-data: filter failed
Traceback (most recent call last):
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\handlers\formhandler.py", line 158, in get
result[key] = yield val
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\tornado\gen.py", line 1133, in run
value = future.result()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\_base.py", line 428, in result
return self.__get_result()
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\_base.py", line 384, in __get_result
raise self._exception
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\concurrent\futures\thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\data.py", line 239, in filter
columns=columns, **kwargs)
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\gramex\data.py", line 1521, in _filter_mongodb
cols = [k for k in table.find().limit(1)[0].keys()]
File "C:\Users\shraddheya.shrivasta\Anaconda3\lib\site-packages\pymongo\cursor.py", line 694, in __getitem__
raise IndexError("no such item for Cursor instance")
IndexError: no such item for Cursor instance
WARNING 19-Feb 00:10:33 web 500 GET /appdata (::1): IndexError('no such item for Cursor instance')
ERROR 19-Feb 00:10:33 __init__ 9988 500 GET /appdata (::1) 846.11ms galaxy-app-data
As of Gramex 1.76.0 (Feb 2022), FormHandler does not connect to empty MongoDB collections, since there is no schema defined.
There is a plan to support empty MongoDB collections by specifying the schema explicitly.
But for now, you should be able to access the collection once you've added at least one row to it.

ClientError 404, cannot connect mongodb to neo4j

Rather than going into painful process of bringing everything into neo4j, we I resorted to easier solution. As configuration is not my expertise https://medium.com/neo4j/transform-mongodb-collections-automagically-into-graphs-9ea085d6e3ef
I followed instructions from here to configure (I want to import data in mongo to neo4j). I am in OSX Catalina. What endpoint should I be using? Following is my settings:
I have a virutal environment created with pipenv.
mongo-connector version: 3.1.1
Python version: 3.9.1
pymongo version: 3.11.3
MongoDB version: 4.4.3
neo4j_doc_manager version: unknown (well installed following instructions from the above).
Now when I try to connect and run following
mongo-connector -m localhost:27017 -t http://localhost:11005/data/db -d neo4j_doc_manager
I used endpoint because I have following endpoints
bolt: https://localhost:11004
http: https://localhost:11005
https: https://localhost:7473
my neo4j location
/Users/Library/Application Support/com.Neo4j.Relate/Data/dbmss/dbms-bbb9318b-9083-4ebf-a934-e17ef055ae22
I have disabled authentication:
dbms.security.auth_enabled=false
I have no idea why I get HTTP 404:
2021-02-05 08:11:15,577 [ALWAYS] mongo_connector.connector:50 - Starting mongo-connector version: 3.1.1
2021-02-05 08:11:15,578 [ALWAYS] mongo_connector.connector:50 - Python version: 3.9.1 (default, Jan 8 2021, 17:17:17)
[Clang 12.0.0 (clang-1200.0.32.28)]
2021-02-05 08:11:15,612 [ALWAYS] mongo_connector.connector:50 - Platform: macOS-10.15.7-x86_64-i386-64bit
2021-02-05 08:11:15,612 [ALWAYS] mongo_connector.connector:50 - pymongo version: 3.11.3
2021-02-05 08:11:15,619 [ALWAYS] mongo_connector.connector:50 - Source MongoDB version: 4.4.3
2021-02-05 08:11:15,619 [ALWAYS] mongo_connector.connector:50 - Target DocManager: mongo_connector.doc_managers.neo4j_doc_manager version: unknown
2021-02-05 08:11:15,633 [CRITICAL] mongo_connector.oplog_manager:713 - Exception during collection dump
Traceback (most recent call last):
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 258, in get
response = self.__base.get(headers=headers, redirect_limit=redirect_limit, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 966, in get
return self.__get_or_head("GET", if_modified_since, headers, redirect_limit, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 943, in __get_or_head
return rq.submit(redirect_limit=redirect_limit, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 452, in submit
return Response.wrap(http, uri, self, rs, **response_kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/packages/httpstream/http.py", line 489, in wrap
raise inst
py2neo.packages.httpstream.http.ClientError: 404 Not Found
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/util.py", line 33, in wrapped
return f(*args, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 78, in bulk_upsert
tx = self.graph.cypher.begin()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 661, in cypher
metadata = self.resource.metadata
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 213, in metadata
self.get()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 267, in get
raise_from(self.error_class(message, **content), error)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/util.py", line 235, in raise_from
raise exception
py2neo.error.GraphError: HTTP GET returned response 404
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/oplog_manager.py", line 668, in do_dump
upsert_all(dm)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/oplog_manager.py", line 651, in upsert_all
dm.bulk_upsert(docs_to_dump(from_coll), mapped_ns, long_ts)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/util.py", line 44, in wrapped
raise new_type(str(exc_value)).with_traceback(exc_tb)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/util.py", line 33, in wrapped
return f(*args, **kwargs)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/mongo_connector/doc_managers/neo4j_doc_manager.py", line 78, in bulk_upsert
tx = self.graph.cypher.begin()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 661, in cypher
metadata = self.resource.metadata
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 213, in metadata
self.get()
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/core.py", line 267, in get
raise_from(self.error_class(message, **content), error)
File "/Users/.local/share/virtualenvs/insiderTrading-L0T4T-gI/lib/python3.9/site-packages/py2neo/util.py", line 235, in raise_from
raise exception
mongo_connector.doc_managers.error_handler.Neo4jOperationFailed: HTTP GET returned response 404
2021-02-05 08:11:15,634 [ERROR] mongo_connector.oplog_manager:723 - OplogThread: Failed during dump collection cannot recover! Collection(Database(MongoClient(host=['localhost:27017'], document_class=dict, tz_aware=False, connect=True, replicaset='rs0'), 'local'), 'oplog.rs')
2021-02-05 08:11:16,630 [ERROR] mongo_connector.connector:408 - MongoConnector: OplogThread <OplogThread(Thread-2, started 123145708253184)> unexpectedly stopped! Shutting down
You should check whether the version of py2neo you are using is compatible with the version of Neo4j. I suspect it isn't, since py2neo.packages.httpstream is a wildly out of date package name, and probably refers too a deprecated/removed HTTP endpoint.

"msg": "not a valid RSA private key file"

I am having issues with the RSA keys when running network_cli for ios.
I am receiving the following error by ansible-playbook:
fatal: [XXX.XXX.XXX.XXX]: FAILED! => {
"msg": "not a valid RSA private key file"
(yml header)
- hosts: [HOSTS]
gather_facts: yes
connection: network_cli
(hosts vars)
[HOSTS:vars]
ansible_network_os=ios
ansible_ssh_private_key_file=/root/.ssh/id_rsa
ansible_ssh_host_keyauto_add=yes
More error details:
The full traceback is:
Traceback (most recent call last):
File "/bin/ansible-connection", line 106, in start
self.connection._connect()
File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/network_cli.py", line 307, in _connect
ssh = self.paramiko_conn._connect()
File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/paramiko_ssh.py", line 249, in _connect
self.ssh = SSH_CONNECTION_CACHE[cache_key] = self._connect_uncached()
File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/paramiko_ssh.py", line 365, in _connect_uncached
raise AnsibleConnectionFailure(msg)
AnsibleConnectionFailure: not a valid RSA private key file

"sam local start-api" fails with "invalid volume specification" on (Windows 10 Pro, Docker for Windows)

I am using SAM CLI 0.6.0 and I am getting the error below when running sam local start-api with the app generated using sam init --runtime java
PS C:\Users\Kiran\AWS\SAM\java-sample\sam-app> sam local start-api
2018-09-03 10:49:49 Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
2018-09-03 10:49:49 You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2018-09-03 10:49:49 * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
2018-09-03 10:49:58 Invoking helloworld.App::handleRequest (java8)
2018-09-03 10:49:58 Found credentials in shared credentials file: ~/.aws/credentials
2018-09-03 10:49:58 Decompressing C:\Users\Kiran\AWS\SAM\java-sample\sam-app\target\HelloWorld-1.0.jar
Fetching lambci/lambda:java8 Docker container image......
2018-09-03 10:49:59 Mounting C:\Users\Kiran\AppData\Local\Temp\tmp7f8z0_zj as /var/task:ro inside runtime container
2018-09-03 10:49:59 Exception on /hello [GET]
Traceback (most recent call last):
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 229, in _raise_for_status
response.raise_for_status()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\requests\models.py", line 939, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://192.168.1.145:2376/v1.35/containers/create
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 2292, in wsgi_app
response = self.full_dispatch_request()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1815, in full_dispatch_request
rv = self.handle_user_exception(e)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1718, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\_compat.py", line 35, in reraise
raise value
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\flask\app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\apigw\local_apigw_service.py", line 140, in _request_handler
self.lambda_runner.invoke(route.function_name, event, stdout=stdout_stream, stderr=self.stderr)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\commands\local\lib\local_lambda.py", line 80, in invoke
self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\lambdafn\runtime.py", line 79, in invoke
self._container_manager.run(container)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\docker\manager.py", line 61, in run
container.create()
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\samcli\local\docker\container.py", line 120, in create
real_container = self.docker_client.containers.create(self._image, **kwargs)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\models\containers.py", line 824, in create
resp = self.client.api.create_container(**create_kwargs)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\container.py", line 411, in create_container
return self.create_container_from_config(config, name)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\container.py", line 422, in create_container_from_config
return self._result(res, True)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 235, in _result
self._raise_for_status(response)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 231, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "C:\Users\Kiran\AppData\Roaming\Python\Python37\site-packages\docker\errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("invalid volume specification: 'C:\Users\Kiran\AppData\Local\Temp\tmp7f8z0_zj:/var/task:ro'")
2018-09-03 10:49:59 127.0.0.1 - - [03/Sep/2018 10:49:59] "GET /hello HTTP/1.1" 502 -
The path mentioned in the last line of the error message (C:\Users\Kiran\AppData\Local\Temp\tmp7f8z0_zj) doesn't seem to exist on my machine. I do have the jar specified code URI of the template file.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
sam-app
Sample SAM Template for sam-app
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 20
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: target/HelloWorld-1.0.jar
Handler: helloworld.App::handleRequest
Runtime: java8
Environment: # More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
Variables:
PARAM1: VALUE
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
Outputs:
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
Appreciate any inputs to fix this error.
Thanks!
Same errors with docker toolbox
Check out this issue.
This is a bug in SAM CLI and fixed with latest release.
Do 'Sam --version' and you should see the following.
sam --version
SAM CLI, version 0.6.1.
This issue affected people running SAM on windows 10 with docker toolbox.
Vyas

Resources