Raw SQL Insert into remote ruby on rails database - ruby-on-rails

I am having trouble with inserting data into an sqlite development database.
My app has 2 servers, one that scrapes browsers (browserscraper) and another that serves client requests. Each of these have a production and development.
I'm setting up development to insert the final scraped data into my development client request server however I can't get the insert to work. I suspect it is related to escaping the content properly but i have been on google for several hours trying to figure this out.
Here is the insert going from my scraping app to my remote client app
#sql_insert = "INSERT INTO #{#table} (`case_number`, `style_of_case`, `circuit`, `judge`, `location`, `disposition`, `date_filed`, `disposition_date`, `case_type`, 'lead_details', 'charge_details')"
#sql_values = " VALUES (#{self.case_number.to_blob}, #{self.style_of_case.to_blob}, #{self.circuit.to_blob}, #{self.judge.to_blob}, #{self.location.to_blob}, #{self.disposition.to_blob}, #{self.date_filed.to_blob}, #{self.disposition_date.to_blob}, #{self.case_type.to_blob}, #{self.lead_details.to_blob}, #{self.charge_details.to_blob});"
#db = SQLite3::Database::new('E:/Sites/aws/db/development.sqlite3')
#db.execute(#sql_insert + #sql_values + "COMMIT;")
The ultimate query looks something like this (quite ugly i know). The last two that i am inserting are yaml
INSERT INTO lead_to_processes (`case_number`, `style_of_case`, `circuit`, `judge`, `location`, `disposition`, `date_filed`, `disposition_date`, `case_type`, 'lead_details', 'charge_details') VALUES (130025129, 130025129 - CITY, 1st(Jim, Counties), LOVEKAMP, KELLY LAREE, Schuyler, Plea Written, 03/19/2012, 03/19/201, Municipal Ordinance - Traffic, ---
1-address_line_1: 6150 RICHLAND RD
1-address_line_2: ''
1-city: 'GEORGIA'
1-birth_year: '1955' 
1-is_alive: 1
, ---
1-Description: Not Available }
1-Code: '95220'
);

You're not hacking PHP in 1999 so you shouldn't be using string interpolation to talk to your database. SQLite3::Database#execute supports placeholders, please use them; your execute should look something like this:
#db.execute("insert into #{#table} (case_number, style_of_case, ...) values (?, ?, ...)", [
self.case_number.to_blob,
self.style_of_case.to_blob,
...
])
That way the database interface will take care of all the quoting and escaping and whatnot for you.

I'm not familiar with Ruby or SQLite, but purely looking at your query you have the last two column names quoted incorrectly with single quotes. 'lead_details' and 'charge_details' should not need to be in quotes unless you use back ticks like the other column names.
Further to that, the values you are inserting are not quoted correctly either. Most languages provide a function to escape and quote database strings appropriately.
I would also suggest checking what the actual error message from your insert is as it should help point you towards the problem in situations like this.

INSERT INTO lead_to_processes (case_number, style_of_case, circuit, judge, location, disposition, date_filed, disposition_date, case_type, 'lead_details', 'charge_details') VALUES (130025129, 130025129 - CITY, 1st(Jim, Counties), LOVEKAMP, KELLY LAREE, Schuyler, Plea Written, 03/19/2012, 03/19/201, Municipal Ordinance - Traffic, --- 1-address_line_1: 6150 RICHLAND RD 1-address_line_2: '' 1-city: 'GEORGIA' 1-birth_year: '1955' 1-is_alive: 1 , --- 1-Description: Not Available } 1-Code: '95220' );
It looks like, starting with 130025129 - CITY, your input values are not surrounded with quotes, so the query parser cannot parse it. I would surround each string value with single quotes.

Related

Run Macro in PhpSpreadsheet

I'm doing a project that allow the customer to export the mysql data into .xls form. I'm using phpspreadsheet library.
That's done, but in my data contain lots of date, some of the date is 0000-00-00 means that it is not used.
I wanted to filter all of these '0000-00-00' into '-'.
I uses excel find and replace and save them as macro ( .bas )
What i have tried is
load the .bas file with IOFactory and reader in php, but it say the file format is not accepted
use substitute method in php loops that use to get the sql data value
$activeSheet->setCellValue('L'.$i, '=substitute('L'.$i ,"0000-00-00", "-')');
$i is 1 that will increase by 1 for each loop
This method failed when the i can't include the $i inside the substitute() because the of "" and
'' problem, I tried to change them around, but seem like the 0000-00-00 and - must use "", if
not the method is not recognise by the library that makes the $i can't be detect then...
Is there any way to solve any of these problems? or it can't be solve in the first place?
cause i can't found any explanation of macro in phpspreadsheet from community nor google.
When setting the value of the cell
if ($datefromselect == '0000-00-00') {
$activeSheet->setCellValueByColumnAndRow($colnum, $rownum, '-');
} else {
$activeSheet->setCellValueByColumnAndRow($colnum, $rownum, $datefromselect);
}
or get it done in the select as in
SELECT lastname,
if(date_closed = '0000-00-00', '-', date_closed)
FROM `lca_clients`

Variable to change label on thingsboard

I would like to see if they can help me with the creation of a variable, where I can change the labels of the MQTT message that is sent from my IoT devices, in order to make it easier and to select the correct parameters when creating a dashboard. .
Example:
This is the message received to my server.
[{"n": "model", "d": "iot-zigbee1783"}, {"n": "Relay", "ap": true}, {"t": "gateway", "ma": "0035DDf45VAIoT215"}]
What I want is to change the label "d" for "deviceIoT" and "ap" for "door sensor" also if it is possible to change the true or false of the door sensor for open and closed.
You can do this with the help of Thingsboards rule-chain.
There is also an official tutorial for this:
https://thingsboard.io/docs/user-guide/rule-engine-2-0/tutorials/transform-incoming-telemetry/
They use the transformation-rule-node called script to convert temperatures from [°F] to [°C].
While this is not your use case, it shows you how to handle incoming telemetry before it is saved to the database.
You could do a mapping of value-keys like this:
var theCustomizedMessage = {};
theCustomizedMessage['customizedKey'] = msg['originalIncomingKey'];
return {msg: theCustomizedMessage, metadata: metadata, msgType: msgType};
Keep in mind that this might be contra-productive since you have to update the rule-node scripts, when something changes.
As an alternative option you can rename the key labels in the widget configuration. This will not help your dashboard developers. But a documentation document will do :)
I strongly recommend against the replacement of boolean values with strings ('closed', 'opened'). This is a job for the widgets (e.g. their value format functions).

How to exclude multiple values in OData call?

I am creating a SAPUI5 application. This application is connected to a backend SAP system via OData. In the SAPUI5 application I use a smart chart control. Out of the box the smart chart lets the user create filters for the underlying data. This works fine - except if you try to use multiple 'not equals' for one property. Is there a way to accomplish this?
I found out that all properties within an 'and_expression' (including nested or_expressions) must have unique name.
The reason why two parameters with the same property don't get parsed into the select options:
/IWCOR/CL_ODATA_EXPR_UTILS=>GET_FILTER_SELECT_OPTIONS takes the expression you pass and parses it into a table of select options.
The select option table returned is of type /IWCOR/IF_ODATA_TYPES=>EDM_SELECT_OPTION_T which is a HASHED TABLE .. WITH UNIQUE KEY property.
From: https://archive.sap.com/discussions/thread/3170195
The problem is that you cannot combine NE terms with OR. Because both parameters after the NE should not be shown in the result set.
So at the end the it_filter_select_options is empty and only the iv_filter_string is filled.
Is there a manual way of facing this problem (evaluation of the iv_filter_string) to handle multiple NE terms?
This would be an example request:
XYZ/SmartChartSet?$filter=(Category%20ne%20%27Smartphone%27%20and%20Category%20ne%20%27Notebook%27)%20and%20Purchaser%20eq%20%27CompanyABC%27%20and%20BuyDate%20eq%20datetime%272018-10-12T02%3a00%3a00%27&$inlinecount=allpages
Normally I want this to exclude items with the category 'Notebook' and 'Smartphone' from my result set that I retrieve from the backend.
If there is a bug inside /iwcor/cl_odata_expr_utils=>get_filter_select_options which makes it unable to treat multiple NE filters of the same component, and you cannot wait for an OSS. I would suggest to wrap it inside a new static method that will make the following logic (if you will be stuck with the ABAP implementation i would try to at least partially implement it when i get time):
Get all instances of <COMPONENT> ne '<VALUE>' inside a () (using REGEX).
Replace each <COMPONENT> with <COMPONENT>_<i> so there will be ( <COMPONENT>_1 ne '<VALUE_1>' and <COMPONENT>_2 ne '<VALUE_2>' and... <COMPONENT>_<n> ne '<VALUE_n>' ).
Call /iwcor/cl_odata_expr_utils=>get_filter_select_options with the modified query.
Modify the rt_select_options result by changing COMPONENT_<i> to <COMPONENT> again.
I can't find the source but I recall that multiple "ne" isn't supported. Isn't that the same thing that happens when you do multiple negatives in SE16, some warning is displayed?
I found this extract for Business ByDesign:
Excluding two values using the OR operator (for example: $filter=CACCDOCTYPE ne ‘1000’ or CACCDOCTYPE ne ‘4000’) is not possible.
The workaround I see is to select the Categories you actively want, not the ones you don't in the UI5 app.
I can also confirm that my code snippet I've used a long time for filtering also has the same problem...
* <SIGNATURE>---------------------------------------------------------------------------------------+
* | Instance Public Method ZCL_MGW_ABS_DATA->FILTERING
* +-------------------------------------------------------------------------------------------------+
* | [--->] IO_TECH_REQUEST_CONTEXT TYPE REF TO /IWBEP/IF_MGW_REQ_ENTITYSET
* | [<-->] CR_ENTITYSET TYPE REF TO DATA
* | [!CX!] /IWBEP/CX_MGW_BUSI_EXCEPTION
* | [!CX!] /IWBEP/CX_MGW_TECH_EXCEPTION
* +--------------------------------------------------------------------------------------</SIGNATURE>
METHOD FILTERING.
FIELD-SYMBOLS <lt_entityset> TYPE STANDARD TABLE.
ASSIGN cr_entityset->* TO <lt_entityset>.
CHECK: cr_entityset IS BOUND,
<lt_entityset> IS ASSIGNED.
DATA(lo_filter) = io_tech_request_context->get_filter( ).
/iwbep/cl_mgw_data_util=>filtering(
exporting it_select_options = lo_filter->get_filter_select_options( )
changing ct_data = <lt_entityset> ).
ENDMETHOD.

Cleaning scraped <a href> rails

I have scraped data from a website and entered it into an array using the code below:
def process_course_details(course_details)
details_array =[]
details_link = true
entry_link = true
details_info = {}
# Sets all data in hash
details_info[:url] = clean_link(course_details.search('div.coursedetails_programmeurl a'))
details_array.push(details_info)
print_details_info(details_info)
entry_link = course_details.search('ul.details_tabs').first
end
The code above stores the element being pulled as such:
View course details on provider's website
But I'd like to clean the above to the below:
http://www.abdn.ac.uk/study/courses/undergraduate/C8R1/
or failing that remove the apostrophe and have this:
View course details on providers website`
You can extract the href with Nokogiri like this:
html = Nokogiri::HTML('View course details on provider\'s website')
html.xpath("//a/#href").to_s # => "http://www.abdn.ac.uk/study/courses/undergraduate/C8R1/"
Based on your comment:
When storing other data I've scraped into the database the apostrophe
provided errors and stopped it. Once I had cleaned the apostrophe and
it no longer was part of the array the code worked and the table was
created.
db = SQLite3::Database.open('ahhh.sqlite3')
db.execute "INSERT INTO aahah (uname, cname, duration, qualification, url, entry) VALUES ('#{#uni_name}', #{#course_name}', '#{#course_duration}', '#{#course_qual}', '#{#details_entry}', '#{#requirements}')"
You are inserting the values via string interpolation:
db.execute("INSERT INTO table_name (foo, bar) VALUES ('#{#foo}', '#{#bar}')")
Apparently, if the interpolated strings contain an apostrophe, your SQL string likely becomes invalid. Even worse, this code is prone to SQL injection.
Instead you should use parameter markers and let the SQLite gem handle the escaping:
db.execute("INSERT INTO table_name (foo, bar) VALUES (?, ?)", [#foo, #bar])
This allows you to safely insert apostrophe and other special characters.

Inserting CrossReference to word via Delphi XE5

Bonjour, Hello,
I'm making this delphi application that I use to create word documents. I'm done with basic word operations (create/save, text, tables..etc).
What I need to do is to insert Page Numbers of headings as Cross Reference in the text. Something like :
"... and the process works as explained on page 23..."
where page number is a hyperlink to a heading. When I recorded a Macros in word it looks like:
Selection.InsertCrossReference ReferenceType:="Heading", _
ReferenceKind:=wdPageNumber, ReferenceItem:="49", InsertAsHyperlink:=True _
, IncludePosition:=False, SeparateNumbers:=False, SeparatorString:=" "
What would be the equivalent in Delphi please?
Thanks in advance!
Arjun.
Even when you're using Late Binding you still need to provide all of the parameters in the same order as the original declaration.
expression.InsertCrossReference(ReferenceType, ReferenceKind, ReferenceItem,
InsertAsHyperlink, IncludePosition, SeparateNumbers, SeparatorString)
If you aren't using a parameter then you can replace it with EmptyParam.
So I think your code will be:
Selection.InsertCrossReference('Heading', 7, '49', True, False, False, ' ');
(I think that wdPageNumber's value is 7).

Resources