after modifying the view as below, if use the view in the select statements by the application(as below query), then we are getting
DB error 5702: The SQL Server is terminating this process.
Here's the query
select *
from view1 v1,
view2 v2
where v1.column in(select M_FLOW_ID from VW_NETPAY_UNDO)
is the issue because more number of view in sql statements ?
Most provably is that your query is runing for a long long time and sybase decided to stop the process, root of the problem ist that you have a cross join between the 2 Views.
You should match the 2 views,(on View1.columnA= View2.columnA)
select *
from view1 v1,
view2 v2 on V1.id = V2.id -- id like common value column
where v1.column in(select colum1 from table1)
.If you don´t do that you have a cartesian products between the 2 Views and then the resuld wouldnt arrise.
Best Regards,
Enrique
Related
We are facing this strange issue with Hive on HDInsight 4.0 - Hive 3.1.0
By default, Hive is set to handle all tables as transactional.
We have 3 tables which join together:
a
b
c
In initial phase 2 of those tables were partitioned by year/month (b,c).
Now, we have repartitioned them by year/month/day (b,c).
Which generates a number of around 200 partitions for each table(b,c).
Now if we do a select from a join b join c we get a transaction lock error.
However, if I do Select a join b - works fine, if I do Select a join c works fine.
Also, if I restrict in the join clause for one of the newly partitioned tables to scan only one partition like
Select a join b on 1=1 join c on 1=1 and c.YEAR=2019 AND c.MONTH=1
it also works fine.
It seems that the increased number of partitions which the join has to scan, or something like that is preventing Hive to take a read lock on those tables... which is very strange, since the tables do not share anything, except the same database.
Any ideas?
Full error:
java.sql.SQLException: Error while processing statement: FAILED: Error in acquiring locks: Error communicating with the metastore
at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266)
at com.hortonworks.hivestudio.hive.HiveJdbcConnectionDelegate.execute(HiveJdbcConnectionDelegate.java:56)
at com.hortonworks.hivestudio.hive.actor.StatementExecutor.runStatement(StatementExecutor.java:93)
at com.hortonworks.hivestudio.hive.actor.StatementExecutor.handleMessage(StatementExecutor.java:74)
at com.hortonworks.hivestudio.hive.actor.HiveActor.onReceive(HiveActor.java:45)
at akka.actor.UntypedAbstractActor$$anonfun$receive$1.applyOrElse(AbstractActor.scala:243)
at akka.actor.Actor.aroundReceive(Actor.scala:514)
at akka.actor.Actor.aroundReceive$(Actor.scala:512)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:132)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
at akka.actor.ActorCell.invoke(ActorCell.scala:496)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
i need to join two table in Qlikview to get result.
Table:
I need to join this two table to get result table like this
Any idea? Can i use cross table and how?
For Table1 you can use CrossTable functionality to "rotate" the table but keeping the first column.
For example:
CrossTable(Location, Quantity)
Load
Reason,
LocA,
LocB
From
[Data.xlsx] (ooxml, embedded labels, table is Table1)
;
The result table after this will be:
Location Reason Quantity
LocA R1 5
LocA R2 4
LocA R3 5
LocA R4 3
LocB R1 2
LocB R2 2
LocB R3 3
LocB R4 5
(you can learn more about CrossTable at Qlik's help site - CrossTable)
After having Table1 in this format you can create composite key (as x3ja suggested). Composite key is basically two (or more) fields concatenated. In your case the join between the tables should be on two fields - Location and Reason.
// CrossTable the data to get it in correct format
Table1_Temp:
CrossTable(Location, Quantity)
Load
Reason,
LocA,
LocB
From
[Data.xlsx] (ooxml, embedded labels, table is Table1)
;
// Resident load to form the composite key
// based on Location and Reason fields
Table1:
Load
Location & '|' & Reason as Key,
Quantity
Resident
Table1_Temp
;
// We dont need Table1_Temp table anymore
Drop Table Table1_Temp;
//Load the second table and create the same composite key
Table2:
Load
Location & '|' & Reason as Key,
Location,
Reason,
Answer
From
[Data.xlsx] (ooxml, embedded labels, table is Table2)
;
After the reload your data model will look like:
And the data:
Notice that the values for Answer, Location, Reason are null in the bottom two rows. This is because the data in Table2 (based on your screenshots) don't contains combination for LocB and R2 and LocA and R4 but Table1 does.
If you want to keep only the combinations that are present in both tables then the approach is similar but with two differences:
Table2 should be loaded first
use keep function to exclude the non common records for being loaded in Table1
(keep at Qlik's help site - keep)
If you want to see the script in action just comment the first tab and uncomment the second one in the example qvw
There are a couple of ways you could do this.
Using association. Load Table 1 twice and concatenate, creating a composite key. So you'd end up with fields ReasonLocation and Quantity. Then load Table 2 creating the same composite key, giving you ReasonLocation, Location, Reason & Answer. Then the tables would associate on that composite key.
Using a join. Load Table1, left join in Table 1 based on Reason with an if statement like if [Location] = 'LocA' then [LocA] else [LocB]. That may need you to load it into a temp table first and do the if statement in a resident load.
You could also combine the two and join the tables in #1 based on the ReasonLocation field.
Hope that helps - sorry it's not fully worked through...
I have a nightly job that runs and computes some data in hive. It is partitioned by day.
Fields:
id bigint
rank bigint
Yesterday
output/dt=2013-10-31
Today
output/dt=2013-11-01
I am trying to figure out if there is a easy way to get incremental changes between today and yesterday
I was thinking about doing a left outer join but not sure what that looks like since its the same table
This is what it might looks like when there are different tables
SELECT * FROM a LEFT OUTER JOIN b
ON (a.id=b.id AND a.dt='2013-11-01' and b.dt='2-13-10-31' ) WHERE a.rank!=B.rank
But on the same table it is
SELECT * FROM a LEFT OUTER JOIN a
ON (a.id=a.id AND a.dt='2013-11-01' and a.dt='2-13-10-31' ) WHERE a.rank!=a.rank
Suggestions?
This would work
SELECT a.*
FROM A a LEFT OUTER JOIN A b ON a.id = b.id
WHERE a.dt='2013-11-01' AND b.dt='2013-10-31' AND <your-rank-conditions>;
Efficiently, this would span 1 MapReduce job only.
So I figured it out... Using Subqueries and Joins
select * from (select * from table where dt='2013-11-01') a
FULL OUTER JOIN
(select * from table where dt='2013-10-31') b
on (a.id=b.id)
where a.rank!=b.rank or a.rank is null or b.rank is null
The above will give you the diff..
You can take the diff and figure out what you need to ADD/UPDATE/REMOVE
UPDATE If a.rank!=null and b.rank!=null i.e rank changed
DELETE IF a.rank=null and b.rank!=null i.e the user is no longer ranked
ADD if a.rank!=null and b.rank=null i.e this is a new user
I have an MVC web site the presents a paged list of data records from a SQL Server database. The UI allows the user to filter the returned data on a number of different criteria, e.g. email address. Here is a snippet of code:
Stopwatch stopwatch = new Stopwatch();
var temp = SubscriberDB
.GetSubscribers(model.Filter, model.PagingInfo);
// Inspect SQL expression here
stopwatch.Start();
model.Subscribers = temp.ToList();
stopwatch.Stop(); // 9 seconds plus compared to < 1 second in Query Analyzer
When this code is run, the StopWatch shows an execution time of around 9 seconds. If I capture the generated SQL expression (just before it is evaluated with the .ToList() method) and cut'n'paste that as a query into SQL Server Management Studio, the execution times drops to less than 1 second. For reference here is the generated SQL expression:
SELECT [t2].[SubscriberId], [t2].[Email], [t3].[Reference] AS [DataSet], [t4].[Reference] AS [DataSource], [t2].[Created]
FROM (
SELECT [t1].[SubscriberId], [t1].[SubscriberDataSetId], [t1].[SubscriberDataSourceId], [t1].[Email], [t1].[Created], [t1].[ROW_NUMBER]
FROM (
SELECT ROW_NUMBER() OVER (ORDER BY [t0].[Email], [t0].[SubscriberDataSetId]) AS [ROW_NUMBER], [t0].[SubscriberId], [t0].[SubscriberDataSetId], [t0].[SubscriberDataSourceId], [t0].[Email], [t0].[Created]
FROM [dbo].[inbox_Subscriber] AS [t0]
WHERE [t0].[Email] LIKE '%_EMAIL_ADDRESS_%'
) AS [t1]
WHERE [t1].[ROW_NUMBER] BETWEEN 0 + 1 AND 0 + 20
) AS [t2]
INNER JOIN [dbo].[inbox_SubscriberDataSet] AS [t3] ON [t3].[SubscriberDataSetId] = [t2].[SubscriberDataSetId]
INNER JOIN [dbo].[inbox_SubscriberDataSource] AS [t4] ON [t4].[SubscriberDataSourceId] = [t2].[SubscriberDataSourceId]
ORDER BY [t2].[ROW_NUMBER]
If I remove the email filter clause, then the controller's StopWatch returns a similar response time to the SQL Management Studio query, less than 1 second - so I am assuming that the basic interface to SQL plumbing is working correctly and that the problem lies with the evaluation of the Linq expression. I should also mention that this is quite a large database with upwards of 1M rows in the subscriber table.
Can anyone throw any light on why there should be such a high (x10) performance differential and what, if anything can be done to address this?
Well not sure about that. 1M rows with a full like can take quiet time. Is Email indexed? Can you run the query with Email% instead of %Email% and see what happen?
I have the following informations and i want to generate an attendance report per day.I will explain the concept
EMP Table
ID NAME
1 Hassan
2 Hussain
3 Hameed
4 Cruz
5 Philip
Transaction table
EmpID Time
1 5/8/2010 8:00 AM
2 5/8/2010 9:00 AM
3 5/8/2010 10:00 AM
My Attendance Report should look like this
Date:5/8/2010
ID NAME STATUS
1 Hassan Present
2 Hussain Present
3 Hameed Present
4 Cruz Absent
5 Philip Absent
I tried the following ,
1.Created a link wth left outer join ,EMP.ID->transcatio.EmpID
2.Group by date from transation table
Created aformula for status,looks like this
If IsNull({transaction.EmpID}) then 'Absent' else 'Present'
But the report displays only those employees,having transactions on that date.
The report looks like this,
ID NAME STATUS
1 Hassan Present
2 Hussain Present
3 Hammed Present
It seeems that,the left outer join didn't work.
Thanks in advance for any help/advice you can give.
Did you look at the actual SQL query that was generated? Go to Database-->Show SQL Query to see it. It may also be helpful to not use a group by date and see what shows up.
Try a right outer join using the same information that will tell you that it did it in the opposite direction of what you wanted.
Have you set the join to be enforced? Stack Overflow Enforced Link Discussion
When you want to join two tables in crystal report with left out join and at the same time want to filter report based on right table e.g. ORDER.ORDER_DATE>='1-JAN-2014' the join will be converted to equi join
to solve the problem some people suggest to use
(ISNULL(ORDER.ORDER_DATE) OR ORDER.RDER_DATE>='1-JAN-2014')
The above solution only works when there are 0 orders for customer and you still want to show those custer but what if some customer have more than 0 orders and orders date is ON or BEFORE '31-DEC-2013'. In such situation ISNULL(ORDER.ORDER_DATE) will not work. to solve such problem you need to either add command object or create a view for orders table as following
Create command object with following sql
Select Customer.customer_name,Order.Order_id, order.order_date
from customer left outer join order on customer.customer_id=order.Customer_id and order.order_date>='1-JAN-2014'
second solution is to create a view for orders as
create or replace view view_orderrs as
Select * from oders where order.order_dt>='1-JAN-2014'
and then use the view in the report instead of order table as the right table.