pyspark drop and merge rows - parsing

I am trying to parse some file and put data to the table:
File = "somehtml.file"
Data = spark.read.text(File)
df_file = Data.select(regexp_extract("col1", '(.*?)', 0).alias("somedata"), \
regexp_extract("col1", '(.*?)', 0).alias("somedata2"))
after that I have not correct result:
+--------------------+--------------------+
| somedata| somedata2|
+--------------------+--------------------+
|http://sweersdsh.ru....| |
| |helo my name lololol...|
| | |
| | |
|http://qweuiewjk.ru....| |
| |helo my name alallal...|
and I needed this one:
+--------------------+--------------------+
| somedata| somedata2|
+--------------------+--------------------+
|http://sweersdsh.ru....|helo my name lololol...|
|http://qweuiewjk.ru....|helo my name alallal...|
this out any '' , please help me

Related

Populate a single google doc using all the data from a google sheet

I am trying to populate a google doc using all the data from a google sheet. What I'm trying to do is to generate a report based on the data in the sheet.
I will be submitting daily report using a google form.
The report will be submitted into the google sheet.
The report recorded in the sheet will be then auto populate the google doc (Only one google doc will be populated).
Currently I have found the way to populate the data in a google doc but it will be generating a new google doc for every row of data.
This is the code that I found
function autofill(e) {
var timestamp = e.values[0];
var name = e.values[1];
var report = e.values[2];
var templateFile = DriveApp.getFileById("Google Doc ID");
var templateResponseFolder = DriveApp.getFolderById("Folder for Google Doc");
var copy = templateFile.makeCopy(name + ',' + report, templateResponseFolder);
var doc = DocumentApp.openById(copy.getId());
var body = doc.getBody();
body.replaceText("{{Timestamp}}", timestamp);
body.replaceText("{{Name}}", name);
body.replaceText("{{Report}}", report);
doc.saveAndClose();
}
Record from Google Sheet
+-----------------+----------+----------+
|Timestamp |Name |Position |
+-----------------+----------+----------+
|3/2/2021 14:12:13|John Doe 1|Position 1|
+-----------------+----------+----------+
|3/2/2021 14:15:13|John Doe 2|Position 2|
+-----------------+----------+----------+
This is the ouput of the above code.
+------------------------+
| John Doe 1 |
| Position 1 |
| |
| |
| |
| |
| |
| |
| |
+------------------------+
John Doe 1, Position 1.docs
+------------------------+
| John Doe 2 |
| Position 2 |
| |
| |
| |
| |
| |
| |
+------------------------+
John Doe 2, Position 2.docs
It will make a new Google Docs document every time a new data is recorded in Google Sheet.
This is what I am trying to achieve
+------------------------+
| John Doe 1 |
| Position 1 |
| |
| |
| |
| |
| |
| Page 1 |
+------------------------+
| John Doe 2 |
| Position 2 |
| |
| |
| |
| |
| |
| Page 2 |
+------------------------+
Report.docs
I am trying to populate all the data recorded in Google Sheet in a single Google Docs which is Report.docs
Modified Script
Since you were calling var copy = templateFile.makeCopy(name + ',' + report, templateResponseFolder); this was making a copy of the file.
With this modification, the templateFile will be updated every time.
function autofill(e) {
var timestamp = e.values[0];
var name = e.values[1];
var report = e.values[2];
var templateFile = DriveApp.getFileById("Google Doc ID");
var body = templateFile.getBody();
body.replaceText("{{Timestamp}}", timestamp);
body.replaceText("{{Name}}", name);
body.replaceText("{{Report}}", report);
doc.saveAndClose();
}
If this is not what you mean, then perhaps you can share a sample spreadsheet with sample data, and what you would like the result to be.
Reference
makeCopy()

Querying in Rails

How can I get the first name and last name from the table User through table Friend which is referenced by the user_id with the condition of user_id 5 and status is pending and place it in a each loop but i also need the contents of friend controller in rails. Thank a lot for the help
Controller
#add = Friend.where(user_id: 5, status: 'Pending')
User Database
+-----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+----------------+
| id | int | NO | PRI | NULL | auto_increment |
| first_name | text | YES | | NULL | |
| last_name | text | YES | | NULL | |
| email | text | YES | | NULL | |
| contact_number | varchar | YES | | NULL | |
| birth_date | date | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| updated_at | datetime | NO | | NULL | |
Friend
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| id | int | NO | PRI | NULL | auto_increment |
| request_date | date | YES | | NULL | |
| reason | text | YES | | NULL | |
| status | varchar(255) | YES | | NULL | |
| userid | int | YES | MUL | NULL | |
How about this:
#add = Friend.joins(:user).select("users.first_name, users.last_name").where(user_id: 5, status: 'Pending')
Seems like youre already fetching the id of the user, stored in the user_id field. In order to get the associated user, i would do
#user = User.find(#add.id)
This retrieves the user based on the id you pass, and from there you can do stuff like
#user.first_name
to get the values
All you need to do in Friend model belongs_to user and then
#add =Friend.where("user_id = ? AND status = ?", 5,"Pending")
and
<#add.each do |s|>
<%s.user.first_name%>
<%=<%s.user.last_name%>
// and so on..
<%end%>

MQL help required, how to generate key

I am struggling with one code line. It is a Key generation Line for a Expert Adviser. Can someone help me figure out how I can generate key by using this line:
int key=3*(StringToInteger(StringSubstr(IntegerToString(AccountNumber()), 0, 3)))+333333;
And what is the problem?
int accountNumber = AccountNumber();
string accountNumberString = IntegerToString(accountNumber);
string accountNumberStringFirst3Digits=
StringSubstr(accountNumberString,0,3);
int accountNumberFirstThreeDigits = StringToInteger(accountNumberStringFirst3Digits);
int accountNumberFirstThreeDigitsMultiplied = 3 * accountNumberFirstThreeDigits;
int key = accountNumberFirstThreeDigitsMultiplied + 333333;
Can someone help me figure out how to generate key by this line?
Welcome, certainly, let's look on that :
int key = 3*(StringToInteger(StringSubstr(IntegerToString(AccountNumber()), 0, 3)))+333333;
Your code actually means this:
// +------------------------------------------------------------------------------- type declaration
// | +--------------------------------------------------------------------------- variable name definition
// | | +------------------------------------------------------------------------- assignment operator
// | | | +----------------------------------------------------------------------- compile-time integer constant
// | | | | +--------------------------------------------------------------------- multiply operator
// | | | | | +-------------------------------------------------- MT4 system function: StringToInteger( aString )
// | | | | | | +------------------------------------ MT4 system function: StringSubstr( aString, aPosToStartSubstrFrom, aSubstrLength )
// | | | | | | | +------------------- MT4 system function: IntegerToString( aIntNum ) | |
// | | | | | | | | +---- MT4 system function: AccountNumber() | |
// | | | | | | | | | | |
// | | | | | | | | | +------------------------------------------------------------------+ |
// | | | | | | | | | | +------------------------------------------------------------------------------+
// | | | | | | | | | | |
int key = 3 * ( StringToInteger( StringSubstr( IntegerToString( AccountNumber() ), 0, 3 ) ) )
+ 333333;
// | ||
// +------||----------------------------------------------------------------- add operator
// +|----------------------------------------------------------------- compile-time integer constant
// +----------------------------------------------------------------- literal MQL4-language syntax-terminator
The code above both defines and generates a fair integer value, so wherever your Expert Advisor code refers to a value of key, this calculated value will be used ( see also the documentation about the New-MQL4 scope-of-validity, inside which this variable remains visible ).

Rail Query includes + filter by column

I start in Ruby and Ruby on rails. I am trying to modify the access to this object (below) in base via the form and I will like to filter by the values that are in the table node_tags.
def map
......
nodes = Node.bbox(bbox).where(:visible => true).includes(:node_tags).limit(MAX_NUMBER_OF_NODES + 1)
.....
end
Probably on SQL request
SELECT * FROM nodes n INNER JOIN node_tags nt on n.node_id = nt.node_id where nt.k = 'alert' and nt.v = 'true'
Table nodes
| node_id | latitude | longitude | changeset_id | visible | timestamp | tile | version | redaction_id |
|---------|-----------|-----------|--------------|---------|-----------|------|---------|--------------|
| 11 | 473705641 | 3955487 | 11 | TRUE |
| 12 | 473705641 | 3955487 | 12 | TRUE |
table node_tags
| node_id | k | v |
|---------|-------|------|
| 11 | name | bob |
| 12 | alert | true |
If I understand right, you need to get sql as in your example by ORM (ActiveRecord).
Try this:
nodes = Node.where(:visible => true).join(:node_tags).where("node_tags.k = 'alert' and node_tags.v = 'true'")

Joining two tables, A and B, where the column names of A should be joined with B based on the values of a column in B

The title is a little confusing, so here is an example of the problem I'm facing.
Table:
FORM_QUESTION
Fields:
student_id
q1_ans
q2_ans
q3_ans
q4_ans
q5_ans
x--------------x----------x----------x----------x----------x----------x
| student_id | q1_ans | q2_ans | q3_ans | q4_ans | q5_ans |
x--------------x----------x----------x----------x----------x----------x
| 1 | A | D | B | B | E |
| 2 | D | C | B | A | D |
| 3 | B | C | D | A | B |
x--------------x----------x----------x----------x----------x----------x
The FORM_QUESTION table stores a student's answers for each question.
Here is information on the second table:
Table:
FORM_VALID_ANS
Fields:
question_id
valid_answer
x---------------x----------------x
| question_id | valid_answer |
x---------------x----------------x
| q1_ans | A |
| q1_ans | B |
| q2_ans | A |
| q2_ans | B |
| q2_ans | C |
| q2_ans | D |
| q3_ans | A |
| q4_ans | A |
| q4_ans | B |
| q5_ans | A |
x---------------x----------------x
The second table, FORM_VALID_ANS, stores the valid, acceptable answers for a particular question. So, according to the above table, here is the acceptable list of values for each question:
q1_ans: A, B
q2_ans: A, B, C, D
q3_ans: A
q4_ans: A, B
q5_ans: A
As you can see, the values for the FORM_VALID_ANS.valid_answer include only the question field names for FORM_QUESTION ("q1_ans", "q2_ans", "q3_ans", "q4_ans", and "q5_ans"). I need to check each students' answer to make sure it is a valid value that they can enter, which is specified in the FORM_VALID_ANS.valid_answer field.
The desired output, where XXX represents an invalid value, and all other values would be answered value:
x--------------x----------x----------x----------x----------x----------x
| student_id | q1_ans | q2_ans | q3_ans | q4_ans | q5_ans |
x--------------x----------x----------x----------x----------x----------x
| 1 | A | D | XXX | B | XXX |
| 2 | XXX | C | XXX | A | XXX |
| 3 | B | C | XXX | A | XXX |
x--------------x----------x----------x----------x----------x----------x
Is it possible to join these two tables together and produce these results (or similar results)?

Resources