Why is this route test failing? - docker

I've been following along with the testdriven.io tutorial for setting up a FastAPI with Docker. The first test I've written using PyTest errored out with the following message:
TypeError: Settings(environment='dev', testing=True, database_url=AnyUrl('postgres://postgres:postgres#web-db:5432/web_test', scheme='postgres', user='*****', password='*****', host='web-db',host_type='int_domain', port='5432', path='/web_test')) is not a callable object.
Looking at the picture, you'll notice that the Settings object has a strange form; in particular, its database_url parameter seems to be wrapping a bunch of other parameters like password, port, and path. However, as shown below my Settings class takes a different form.
From config.py:
# ...imports
class Settings(BaseSettings):
environment: str = os.getenv("ENVIRONMENT", "dev")
testing: bool = os.getenv("TESTING", 0)
database_url: AnyUrl = os.environ.get("DATABASE_URL")
#lru_cache()
def get_settings() -> BaseSettings:
log.info("Loading config settings from the environment...")
return Settings()
Then, in the conftest.py module, I've overridden the settings above with the following:
import os
import pytest
from fastapi.testclient import TestClient
from app.main import create_application
from app.config import get_settings, Settings
def get_settings_override():
return Settings(testing=1, database_url=os.environ.get("DATABASE_TEST_URL"))
#pytest.fixture(scope="module")
def test_app():
app = create_application()
app.dependency_overrides[get_settings] = get_settings_override()
with TestClient(app) as test_client:
yield test_client
As for the offending test itself, that looks like the following:
def test_ping(test_app):
response = test_app.get("/ping")
assert response.status_code == 200
assert response.json() == {"environment": "dev", "ping": "pong", "testing": True}
The container is successfully running on my localhost without issue; this leads me to believe that the issue is wholly related to how I've set up the test and its associated config. However, the structure of the error and how database_url is wrapping up all these key-value pairs from docker-compose.yml gives me the sense that my syntax error could be elsewhere.
At this juncture, I'm not sure if the issue has something to do with how I set up test_ping.py, my construction of the settings_override, with the format of my docker-compose.yml file, or something else altogether.
So far, I've tried to fix this issue by reading up on the use of dependency overrides in FastApi, noodling with my indentation in the docker-compose, changing the TestClient from one provided by starlette to that provided by FastAPI, and manually entering testing mode.
Something I noticed when attempting to manually go into testing mode was that the container doesn't want to follow suit. I've tried setting testing to 1 in docker-compose.yml, and testing: bool = True in config.Settings.
I'm new to all of the relevant tech here and bamboozled. What is causing this discrepancy with my test? Any and all insight would be greatly appreciated. If you need to see any other files, or are interested in the package structure, just let me know. Many thanks.

Any dependency override through app.dependency_overrides should provide the function being overridden as the key and the function that should be used instead. In your case you're assigning the correct override, but you're assigning the result of the function as the override, and not the override itself:
app.dependency_overrides[get_settings] = get_settings_override()
.. this should be:
app.dependency_overrides[get_settings] = get_settings_override
The error message shows that FastAPI tried to call your dictionary as a function, something that hints to it expecting a function instead.

Related

How do you authenticate with an API key inside an IBM Cloud Function?

I am writing an IBM Cloud Function which uses the python SDK to interface with a Cloudant service. I have the Cloudant service up, the databases populated, and service credentials / API key ready. However when I try to instantiate the CloudantV1 service inside my Function I get a runtime error "must provide authenticator".
I looked up the error in their git repos and it seems like it is trying to setup an authenticator object by looking up values from environment variables, which do not exist in the Function. I just want to pass my API key directly, but I have not found a method to do this. I am using basic code from the examples so I think my calls are correct.
I have considered injecting the environment variables inside the Function, but that sounds like a major hack. I must be doing something incorrectly. Please help me understand what it is. Here is basic Function python code which reproduces the error:
from ibmcloudant.cloudant_v1 import CloudantV1
def main(params_dict):
service = CloudantV1.new_instance()
# unreachable
return { "message" : "hello world" }
There is an example for programmatic authentication at https://cloud.ibm.com/apidocs/cloudant?code=python#programmatic-authentication - it basically looks like this:
from ibmcloudant.cloudant_v1 import CloudantV1
from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
authenticator = IAMAuthenticator('yourAPIkey')
service = CloudantV1(authenticator=authenticator)
service.set_service_url('https://yourserviceurl.example')

Luigi: Running Tasks in a Docker Image

I am testing Luigi's abilities for running tasks in a Docker container, i.e. I would like Luigi to spawn a container from a given Docker image and execute a task therein.
As far as I understood, there is luigi.contrib.docker_runner.DockerTask for this purpose. I tried to modify its command and add an output:
import luigi
from luigi.contrib.docker_runner import DockerTask
class Task(DockerTask):
def output(self):
return luigi.LocalTarget("bla.txt")
def command(self):
return f"touch {self.output()}"
if __name__ == "__main__":
luigi.build([Task()], workers=2, local_scheduler=True)
But I am getting
It seems that there is a TypeError in docker. Is my use of DockerTask erroneous? Unfortunately, I cannot find any examples for use cases...
Here the answer from my reply to your email:
luigi.LocalTarget doesn't return a file path or string, but a target object, therefore f"touch {self.output()}" isn't a valid shell command.
Second, I just looked into the documentation of DockerTask.
https://luigi.readthedocs.io/en/stable/_modules/luigi/contrib/docker_runner.html#DockerTask
command is should be a property, not a method. This explains your error message, the code calls "Task().command", which is a function, and not a string. A solution might be to use a static attribute for the output name
class Task(DockerTask):
output_name = "bla.txt"
def output(self):
return luigi.LocalTarget(self.output_name)
command = f"touch {self.output_name}"
or to use a property decorator:
class Task(DockerTask):
def output(self):
return luigi.LocalTarget("bla.txt")
#property
def command(self):
target = self.output() # not a string/fpath
return f"touch {target.path}"
though I'm not sure whether this would work, I'd prefer the first option.
In the future, make sure to check the documentation for the base class that you're implementing, e.g. checking if an attribute is a method or property, maybe use a debugger to understand what's happening.
And for the sake of searchability and saving data and bandwidth, in the future try to post error logs not as screenshots but in text, either as a snippet or via a link to a pastebin or something like that.

import python functions into serversless

I h got an issue with the following serverless config.
this is my handler and the files/folders structure.
the issue is that after uploading my project to AWS when I test my lambda I got an error as follows:
lambda execution fails: "errorMessage": "Unable to import module 'app_monitor': No module named 'monitoring'"
{
"errorMessage": "Unable to import module 'src/app_monitor': No module named 'monitoring'",
"errorType": "Runtime.ImportModuleError",
"requestId": "bca3f67d-815f-452b-a2a6-c713ad2c6baa",
"stackTrace": []
}
have you got any clue how can I add this into serverless config.?
First, a quick tip on troubleshooting: When I ran into such issues it was helpful to go to the AWS console, look at the lambda function, and see what the uploaded file structure looks like on that end. Is the monitoring folder there?
Moreover, in order to specify how a specific function is packaged, you have to explicitly state that you want it to be individually packaged and not follow the general rules of the project as a whole.
You should try to add:
app_monitoring:
package:
individually: true
patterns:
- 'src/**'
More documentation on packaging configuration here
You may also have better luck with explicitly stating the patterns you need, I know I've had issues with globs in the past. So for example you can try:
patterns:
- 'src/app_monitoring.py'
- 'src/monitoring/get_lb.py'

Failures in init.groovy.d scripts: null values returned

I'm trying to get Jenkins set up, with configuration, within a Docker environment. Per a variety of sources, it appears the suggested method is to insert scripts into JENKINS_HOME/init.groovy.d. I've taken scripts from places like the Jenkins wiki and made slight changes. They're only partially working. Here is one of them:
import java.util.logging.ConsoleHandler
import java.util.logging.FileHandler
import java.util.logging.SimpleFormatter
import java.util.logging.LogManager
import jenkins.model.Jenkins
// Log into a file
println("extralogging.groovy")
def RunLogger = LogManager.getLogManager().getLogger("hudson.model.Run")
def logsDir = new File("/var/log/jenkins")
if (!logsDir.exists()) { logsDir.mkdirs() }
FileHandler handler = new FileHandler(logsDir.absolutePath+"/jenkins-%g.log", 1024 * 1024, 10, true);
handler.setFormatter(new SimpleFormatter());
RunLogger.addHandler(handler)
This script fails on the last line, RunLogger.addHandler(handler).
2019-12-20 19:25:18.231+0000 [id=30] WARNING j.util.groovy.GroovyHookScript#execute: Failed to run script file:/var/lib/jenkins/init.groovy.d/02-extralogging.groovy
java.lang.NullPointerException: Cannot invoke method addHandler() on null object
I've had a number of other scripts return NULL objects from various gets similar to this one:
def RunLogger = LogManager.getLogManager().getLogger("hudson.model.Run")
My goal is to be able to develop (locally) a Jenkins implementation and then hand it to our sysops guys. Later, as I add pipelines and what not, I'd like to be able to also work on them in a local Jenkins configuration and then hand something for import into production Jenkins.
I'm not sure how to produce API documentation so I can chase this myself. Maybe I need to stop doing it this way and just grab the files that get modified when I do this via the GUI and just stuff the files into the right place.
Suggestions?

Equivalence of Rails console for Node.js

I am trying out Node.js Express framework, and looking for plugin that allows me to interact with my models via console, similar to Rails console. Is there such a thing in NodeJS world?
If not, how can I interact with my Node.js models and data, such as manually add/remove objects, test methods on data etc.?
Create your own REPL by making a js file (ie: console.js) with the following lines/components:
Require node's built-in repl: var repl = require("repl");
Load in all your key variables like db, any libraries you swear by, etc.
Load the repl by using var replServer = repl.start({});
Attach the repl to your key variables with replServer.context.<your_variable_names_here> = <your_variable_names_here>. This makes the variable available/usable in the REPL (node console).
For example: If you have the following line in your node app:
var db = require('./models/db')
Add the following lines to your console.js
var db = require('./models/db');
replServer.context.db = db;
Run your console with the command node console.js
Your console.js file should look something like this:
var repl = require("repl");
var epa = require("epa");
var db = require("db");
// connect to database
db.connect(epa.mongo, function(err){
if (err){ throw err; }
// open the repl session
var replServer = repl.start({});
// attach modules to the repl context
replServer.context.epa = epa;
replServer.context.db = db;
});
You can even customize your prompt like this:
var replServer = repl.start({
prompt: "Node Console > ",
});
For the full setup and more details, check out:
http://derickbailey.com/2014/07/02/build-your-own-app-specific-repl-for-your-nodejs-app/
For the full list of options you can pass the repl like prompt, color, etc: https://nodejs.org/api/repl.html#repl_repl_start_options
Thank you to Derick Bailey for this info.
UPDATE:
GavinBelson has a great recommendation for running with sequelize ORM (or anything that requires promise handling in the repl).
I am now running sequelize as well, and for my node console I'm adding the --experimental-repl-await flag.
It's a lot to type in every time, so I highly suggest adding:
"console": "node --experimental-repl-await ./console.js"
to the scripts section in your package.json so you can just run:
npm run console
and not have to type the whole thing out.
Then you can handle promises without getting errors, like this:
const product = await Product.findOne({ where: { id: 1 });
I am not very experienced in using node, but you can enter node in the command line to get to the node console. I then used to require the models manually
Here is the way to do it, with SQL databases:
Install and use Sequelize, it is Node's ORM answer to Active Record in Rails. It even has a CLI for scaffolding models and migrations.
node --experimental-repl-await
> models = require('./models');
> User = models.User; //however you load the model in your actual app this may vary
> await User.findAll(); //use await, then any sequelize calls here
TLDR
This gives you access to all of the models just as you would in Rails active record. Sequelize takes a bit of getting used to, but in many ways it is actually more flexible than Active Record while still having the same features.
Sequelize uses promises, so to run these properly in REPL you will want to use the --experimental-repl-await flag when running node. Otherwise, you can get bluebird promise errors
If you don't want to type out the require('./models') step, you can use console.js - a setup file for REPL at the root of your directory - to preload this. However, I find it easier to just type this one line out in REPL
It's simple: add a REPL to your program
This may not fully answer your question, but to clarify, node.js is much lower-level than Rails, and as such doesn't prescribe tools and data models like Rails. It's more of a platform than a framework.
If you are looking for a more Rails-like experience, you may want to look at a more 'full-featured' framework built on top of node.js, such as Meteor, etc.

Resources