how to pass additional parametes to spider parse function in scrapy during web scraping - parsing

I am trying to pass additional information to the parse function but it is giving a type error.
TypeError: parse() got an unexpected keyword argument 'body'
i am unable to resolve this issue.
"""
return [scrapy.Request(url=website.search_url.format(prod), callback=self.parse,
cb_kwargs = {"body":website.body_xpath,"product_list":website.products_list_xpath,
"names":website.products_name_xpath,"selling_price":website.selling_price_xpath,
"market_price":website.market_price_xpath}) for website in websites for prod in modified_products]
def parse(self, response):
body = response.cb_kwargs.get("body")
product_list = response.cb_kwargs.get("product_list")
name = response.cb_kwargs.get("names")
selling_price = response.cb_kwargs.get("selling_price")
market_price = response.cb_kwargs.get("market_price")
"""

I forgot to write those names in parse function definition, after adding them i am getting the correct result. Thanks for having a look at it.
"""
return [scrapy.Request(url=website.search_url.format(prod), callback=self.parse,
cb_kwargs = dict(body = website.body_xpath, product_list = website.products_list_xpath,
name = website.products_name_xpath, selling_price = website.selling_price_xpath,
market_price = website.market_price_xpath)) for website in websites for prod in modified_products]
def parse(self, response, body, product_list, name, selling_price, market_price):
body = response.cb_kwargs["body"]
product_list = response.cb_kwargs["product_list"]
name_ = response.cb_kwargs["name"]
selling_price_ = response.cb_kwargs["selling_price"]
market_price_ = response.cb_kwargs["market_price"]
"""

Related

Upload Offline Conversion migration to V9

I used to upload offline conversion using following code in v201809 version as provided at
https://github.com/googleads/googleads-php-lib/blob/master/examples/AdWords/v201809/Remarketing/UploadOfflineConversions.php
$oAuth2Credential = (new OAuth2TokenBuilder())->fromFile()->build();
$session = (new AdWordsSessionBuilder())->fromFile()->withOAuth2Credential($oAuth2Credential)->withClientCustomerId($customerid)->enablePartialFailure()->build();
$adWordsServices = new AdWordsServices();
$offlineConversionService = $adWordsServices->get($session, OfflineConversionFeedService::class);
$conversionName="OfflineConv";
$feed = new OfflineConversionFeed();
$feed->setConversionName($conversionName);
$feed->setConversionTime($conversionTime);
$feed->setConversionValue($conversionValue);
$feed->setGoogleClickId($gclid);
$offlineConversionOperation = new OfflineConversionFeedOperation();
$offlineConversionOperation->setOperator(Operator::ADD);
$offlineConversionOperation->setOperand($feed);
$offlineConversionOperations = [$offlineConversionOperation];
$result = $offlineConversionService->mutate($offlineConversionOperations);
Now I am upgrading to V9, I have used the code as provided at
https://github.com/googleads/google-ads-php/blob/main/examples/Remarketing/UploadOfflineConversion.php
$oAuth2Credential = (new OAuth2TokenBuilder())->fromFile()->build();
$googleAdsClient = (new GoogleAdsClientBuilder())->fromFile()->withOAuth2Credential($oAuth2Credential)->build();
//$conversionName="OfflineConv";
$conversionName = ConversionActionType::WEBPAGE;
$clickConversion = new ClickConversion([
'conversion_action' => ResourceNames::forConversionAction($customerId, $conversionName),
'gclid' => $gclid,
'conversion_value' => $conversionValue,
'conversion_date_time' => $conversionTime,
'currency_code' => 'USD'
]);
$conversionUploadServiceClient = $googleAdsClient->getConversionUploadServiceClient();
$result = $conversionUploadServiceClient->uploadClickConversions($customerid, [$clickConversion], true);
The problem is when we set $conversionName="OfflineConv"; we get following error.
Resource name 'customers/9025381111/conversionActions/OfflineConv' is malformed: expected 'customers/{customer_id}/conversionActions/{ConversionType.conversion_type_id}'., at conversions[0].conversion_action
and when we set $conversionName = ConversionActionType::WEBPAGE; we get following error.
This customer does not have an import conversion action that matches the conversion action provided., at conversions[0].conversion_action
Can someone help me?
Conversion Name must match the conversion action that you've already set up in your account. You're passing the enum value for the type.
It should be something like $conversionName = "OfflineConversions"
where "OfflineConversions" is exactly the name of the conversion in the conversions section of the web UI.
ConversionName has been removed from V9, use $conversionActionId as in example on github.
You need to try use parameter ctId from url your "OfflineConversions" in Google Ads UI as $conversionActionId.
Or try the solution from here.
For your example, replace this
$conversionName = ConversionActionType::WEBPAGE;
on this
$conversionName = {ctId from url};
You need to first create a click conversion
click_conversion = adwords_client.get_type("ClickConversion")
conversion_action_service = adwords_client.get_service('ConversionActionService')
click_conversion.conversion_action = (
conversion_action_service.conversion_action_path(
customer_id, conversion_action_id
)
)
The conversion_action_id is equivalent to conversion_name of previous version.
You can find the id using following snippet
ads: GoogleAdsServiceClient = self.adwords_client.get_service('GoogleAdsService')
pages = ads.search(query="SELECT conversion_action.id, conversion_action.name FROM conversion_action where conversion_action.name={conversion_name}", customer_id={customer_id})
for page in pages:
print(page.conversion_action)
Then you upload the conversion
click_conversion.gclid = gclid
click_conversion.conversion_value = conversion_value
click_conversion.conversion_date_time = conversion_time
click_conversion.currency_code = currency_code
conversion_upload_service = self.adwords_client.get_service("ConversionUploadService")
request = self.adwords_client.get_type("UploadClickConversionsRequest")
request.customer_id = customer_id
request.conversions.append(click_conversion)
request.partial_failure = True
conversion_upload_response = (
conversion_upload_service.upload_click_conversions(
request=request,
)
)
uploaded_click_conversion = conversion_upload_response.results[0]
print(
f"Uploaded conversion that occurred at "
f'"{uploaded_click_conversion.conversion_date_time}" from '
f'Google Click ID "{uploaded_click_conversion.gclid}" '
f'to "{uploaded_click_conversion.conversion_action}"'
)

?:0: attempt to perform arithmetic on field 'fileSize' (a nil value)

How to resolved error or possible ways to resolved it?
Guys, i've developed plugin using Lua language which can be integrate or run from Adobe's LightRoom Classic. Currently i need to upload or send a file to server but i can not. Everytime i called the POST API which is multipart/form-data error popup "?:0: attempt to perform arithmetic on field 'fileSize' (a nil value)". Not even API is being called this error pops up before API call. after debug I can assure the possible issue is in creating mimeChunks with file type.
I have developed the code like below, can any one help me out with suggestions so that i can able to resolved issue?
local filePath = assert("C:\Users\Ankit\Desktop\Hangman.PNG")
local fileName = LrPathUtils.leafName(filePath)
local mimeChunks = {}
mimeChunks[#mimeChunks + 1] = {
name = 'api_sig',
value = "test value"
}
mimeChunks[#mimeChunks + 1] = {
name = "file",
filePath = filePath,
fileName = fileName,
contentType = "application/octet-stream"
}
local postUrl = "API endpoint"
local result, hdrs = LrHttp.postMultipart(postUrl, mimeChunks)
if result then
LrDialogs.message("Form Values", result)
else
LrDialogs.message("Form Values", "API issue")
end
Eventually image or file path itself cause the issue, there are no such indications or articles related to this functionality, but yes "add-on backslash" will work out for sure. Kindly review the below code for more detailed bifurcation which pass dynamic selected file or image path.
local function uploadFile(filePath)
local fileName = LrPathUtils.leafName( filePath )
local mimeChunks = {}
mimeChunks[ #mimeChunks + 1 ] = { name = 'api_sig', value = "test value"}
mimeChunks[#mimeChunks + 1] = {
name = "file",
filePath = filePath,
fileName = fileName,
contentType = "image/jpeg" --multipart/form-data --application/octet-stream
}
import "LrTasks".startAsyncTask(
function()
local postUrl = "http://cms.local.com/api/v1/upload"
local result, hdrs = LrHttp.postMultipart(postUrl, mimeChunks)
if result then
LrDialogs.message("Image uploaded.", result)
else
LrDialogs.message("Error", "API issue")
end
end
)
end
Above uploadFile method will automatically call the API and post form-data collection. Below code is for call uploadFile function which select all the images from catalog.
for p, photo in ipairs(LrApplication.activeCatalog()) do
uploadFile(assert(photo:getRawMetadata('path')));
end
Above code will help you out the selection of categlog with Adobe's LightRoom Plugin.

Springdoc-openapi external json file with relative path for example request body

I am looking for a way to use a relative path for external JSON files while defining the example request body.
My current definition is like this:
#Operation(requestBody = #RequestBody(description = "Request", content = #Content(schema = #Schema(implementation = Request.class), examples = {
#ExampleObject(
name = "An example request",
value = "{\n" +
"\"token\":\"token\"\n" +
"}",
summary = "Request"
)}
)),summary = "summ", description = "desc")
I want to keep JSON value on an external JSON file.
I know that "externalValue" field can be used instead of "value" to reference an external json file, but it requires a URL. I want to keep it in the project file and use the relative path. I am using Open API v3.
For those who are just looking for external JSON file usage, you can use external JSON file from a URL like this:
#Operation(requestBody = #RequestBody(description = "Request", content = #Content(schema = #Schema(implementation = Request.class), examples = {
#ExampleObject(
name = "An example request",
externalValue = "http://domain/test/example.json",
summary = "Request"
)}
)),summary = "summ", description = "desc")

Cannot parse response body content in katalon studio

I’m facing a problem, where I can’t parse response body content.
Here is what I use for parsing, that works for another responses but for current response it doesn’t work.
String getContent = get_response.getResponseBodyContent()
JsonSlurper slurper = new JsonSlurper()
Map parsedJson = slurper.parseText(getContent)
And it gives me a following error:
This is because you have a JSON array in your response body content. Try this one:
List parsedJson = slurper.parseText(getContent)
or just
def parsedJson = slurper.parseText(getContent)
Detailed example:
def json = """
[
{
"companyName":"Foo",
"customerId":"Bar"
},
{
"companyName":"Foo2",
"customerId":"Bar2"
}
]
"""
def slurper = new JsonSlurper()
//Map mapJson = slurper.parseText(json) FAIL!!!
List listJson = slurper.parseText(json)
def objJson = slurper.parseText(json)
objJson.each { map ->
println(map)
}
Output:
[companyName:Foo, customerId:Bar]
[companyName:Foo2, customerId:Bar2]

Send HTTPService Request in flex 3 with '-' in the URl Paramerters to get Google Feeds

I am developing application in flex 3 which interacts with the Google feeds to produce my results. The URL to which i want to send request is something like this
http://books.google.com/books/feeds/volumes?q=football+-soccer&start-index=11&max-results=10
Now i can send and receive results with q parameter, but in the next two parameters has a '-' (start-index and max-results). I am using HTTPService to send the requeset like this.
SearchService.url = "http://books.google.com/books/feeds/volumes";
SearchService.method = "GET";
SearchService.contentType = "application/x-www-form-urlencoded"
Here SearchService is the HTTPService
var params:Object = new Object();
params.q = searchText;
params.start-index = 11;
params.max-results = 100;
service.SearchService.send(params);
Now my flex IDE throws me a error stating 'Cannot assign a non-reference value'. Only if i send the request with this parameters, i could put pagination in my application. So how can i send HTTPService request with '-' in the URL parameters?
You can do:
var params:Object = new Object();
params["q"] = searchText;
params["start-index"] = 11;
params["max-results"] = 100;
service.SearchService.send(params);
Validated and tested to work properly! :)

Resources