I have created the model and save the weights using google colab. Now I have created a prediction script.
The prediction script contains the model class. I am trying to load the model weights using the following method-
Saving & Loading Model Across Devices
Save on GPU, Load on CPU
Save:
torch.save(model.state_dict(), PATH)
Load:
device = torch.device('cpu')
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH, map_location=device))
The above method should work, right? Yes.
But when I am trying to do so I have different parameters of the model in Google Colab (Prediction, runtime-None, device=CPU) and different in my local machine (prediction, device=cpu)
Model Params in Colab-
def count_parameters(model):
return sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f'The model has {count_parameters(model):,} trainable parameters')
The model has 12,490,234 trainable parameters
+-------------------------------------------------------+------------+
| Modules | Parameters |
+-------------------------------------------------------+------------+
| encoder.tok_embedding.weight | 2053376 |
| encoder.pos_embedding.weight | 25600 |
| encoder.layers.0.self_attn_layer_norm.weight | 256 |
| encoder.layers.0.self_attn_layer_norm.bias | 256 |
| encoder.layers.0.ff_layer_norm.weight | 256 |
| encoder.layers.0.ff_layer_norm.bias | 256 |
| encoder.layers.0.self_attention.fc_q.weight | 65536 |
| encoder.layers.0.self_attention.fc_q.bias | 256 |
| encoder.layers.0.self_attention.fc_k.weight | 65536 |
| encoder.layers.0.self_attention.fc_k.bias | 256 |
| encoder.layers.0.self_attention.fc_v.weight | 65536 |
| encoder.layers.0.self_attention.fc_v.bias | 256 |
| encoder.layers.0.self_attention.fc_o.weight | 65536 |
| encoder.layers.0.self_attention.fc_o.bias | 256 |
| encoder.layers.0.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.0.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.0.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.0.positionwise_feedforward.fc_2.bias | 256 |
| encoder.layers.1.self_attn_layer_norm.weight | 256 |
| encoder.layers.1.self_attn_layer_norm.bias | 256 |
| encoder.layers.1.ff_layer_norm.weight | 256 |
| encoder.layers.1.ff_layer_norm.bias | 256 |
| encoder.layers.1.self_attention.fc_q.weight | 65536 |
| encoder.layers.1.self_attention.fc_q.bias | 256 |
| encoder.layers.1.self_attention.fc_k.weight | 65536 |
| encoder.layers.1.self_attention.fc_k.bias | 256 |
| encoder.layers.1.self_attention.fc_v.weight | 65536 |
| encoder.layers.1.self_attention.fc_v.bias | 256 |
| encoder.layers.1.self_attention.fc_o.weight | 65536 |
| encoder.layers.1.self_attention.fc_o.bias | 256 |
| encoder.layers.1.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.1.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.1.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.1.positionwise_feedforward.fc_2.bias | 256 |
| encoder.layers.2.self_attn_layer_norm.weight | 256 |
| encoder.layers.2.self_attn_layer_norm.bias | 256 |
| encoder.layers.2.ff_layer_norm.weight | 256 |
| encoder.layers.2.ff_layer_norm.bias | 256 |
| encoder.layers.2.self_attention.fc_q.weight | 65536 |
| encoder.layers.2.self_attention.fc_q.bias | 256 |
| encoder.layers.2.self_attention.fc_k.weight | 65536 |
| encoder.layers.2.self_attention.fc_k.bias | 256 |
| encoder.layers.2.self_attention.fc_v.weight | 65536 |
| encoder.layers.2.self_attention.fc_v.bias | 256 |
| encoder.layers.2.self_attention.fc_o.weight | 65536 |
| encoder.layers.2.self_attention.fc_o.bias | 256 |
| encoder.layers.2.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.2.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.2.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.2.positionwise_feedforward.fc_2.bias | 256 |
| decoder.tok_embedding.weight | 3209728 |
| decoder.pos_embedding.weight | 25600 |
| decoder.layers.0.self_attn_layer_norm.weight | 256 |
| decoder.layers.0.self_attn_layer_norm.bias | 256 |
| decoder.layers.0.enc_attn_layer_norm.weight | 256 |
| decoder.layers.0.enc_attn_layer_norm.bias | 256 |
| decoder.layers.0.ff_layer_norm.weight | 256 |
| decoder.layers.0.ff_layer_norm.bias | 256 |
| decoder.layers.0.self_attention.fc_q.weight | 65536 |
| decoder.layers.0.self_attention.fc_q.bias | 256 |
| decoder.layers.0.self_attention.fc_k.weight | 65536 |
| decoder.layers.0.self_attention.fc_k.bias | 256 |
| decoder.layers.0.self_attention.fc_v.weight | 65536 |
| decoder.layers.0.self_attention.fc_v.bias | 256 |
| decoder.layers.0.self_attention.fc_o.weight | 65536 |
| decoder.layers.0.self_attention.fc_o.bias | 256 |
| decoder.layers.0.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_q.bias | 256 |
| decoder.layers.0.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_k.bias | 256 |
| decoder.layers.0.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_v.bias | 256 |
| decoder.layers.0.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_o.bias | 256 |
| decoder.layers.0.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.0.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.0.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.0.positionwise_feedforward.fc_2.bias | 256 |
| decoder.layers.1.self_attn_layer_norm.weight | 256 |
| decoder.layers.1.self_attn_layer_norm.bias | 256 |
| decoder.layers.1.enc_attn_layer_norm.weight | 256 |
| decoder.layers.1.enc_attn_layer_norm.bias | 256 |
| decoder.layers.1.ff_layer_norm.weight | 256 |
| decoder.layers.1.ff_layer_norm.bias | 256 |
| decoder.layers.1.self_attention.fc_q.weight | 65536 |
| decoder.layers.1.self_attention.fc_q.bias | 256 |
| decoder.layers.1.self_attention.fc_k.weight | 65536 |
| decoder.layers.1.self_attention.fc_k.bias | 256 |
| decoder.layers.1.self_attention.fc_v.weight | 65536 |
| decoder.layers.1.self_attention.fc_v.bias | 256 |
| decoder.layers.1.self_attention.fc_o.weight | 65536 |
| decoder.layers.1.self_attention.fc_o.bias | 256 |
| decoder.layers.1.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_q.bias | 256 |
| decoder.layers.1.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_k.bias | 256 |
| decoder.layers.1.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_v.bias | 256 |
| decoder.layers.1.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_o.bias | 256 |
| decoder.layers.1.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.1.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.1.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.1.positionwise_feedforward.fc_2.bias | 256 |
| decoder.layers.2.self_attn_layer_norm.weight | 256 |
| decoder.layers.2.self_attn_layer_norm.bias | 256 |
| decoder.layers.2.enc_attn_layer_norm.weight | 256 |
| decoder.layers.2.enc_attn_layer_norm.bias | 256 |
| decoder.layers.2.ff_layer_norm.weight | 256 |
| decoder.layers.2.ff_layer_norm.bias | 256 |
| decoder.layers.2.self_attention.fc_q.weight | 65536 |
| decoder.layers.2.self_attention.fc_q.bias | 256 |
| decoder.layers.2.self_attention.fc_k.weight | 65536 |
| decoder.layers.2.self_attention.fc_k.bias | 256 |
| decoder.layers.2.self_attention.fc_v.weight | 65536 |
| decoder.layers.2.self_attention.fc_v.bias | 256 |
| decoder.layers.2.self_attention.fc_o.weight | 65536 |
| decoder.layers.2.self_attention.fc_o.bias | 256 |
| decoder.layers.2.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_q.bias | 256 |
| decoder.layers.2.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_k.bias | 256 |
| decoder.layers.2.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_v.bias | 256 |
| decoder.layers.2.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_o.bias | 256 |
| decoder.layers.2.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.2.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.2.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.2.positionwise_feedforward.fc_2.bias | 256 |
| decoder.fc_out.weight | 3209728 |
| decoder.fc_out.bias | 12538 |
+-------------------------------------------------------+------------+
Total Trainable Params: 12490234
Model Params in Local-
def count_parameters(model):
return sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f'The model has {count_parameters(model):,} trainable parameters')
The model has 12,506,137 trainable parameters
+-------------------------------------------------------+------------+
| Modules | Parameters |
+-------------------------------------------------------+------------+
| encoder.tok_embedding.weight | 2053376 |
| encoder.pos_embedding.weight | 25600 |
| encoder.layers.0.self_attn_layer_norm.weight | 256 |
| encoder.layers.0.self_attn_layer_norm.bias | 256 |
| encoder.layers.0.ff_layer_norm.weight | 256 |
| encoder.layers.0.ff_layer_norm.bias | 256 |
| encoder.layers.0.self_attention.fc_q.weight | 65536 |
| encoder.layers.0.self_attention.fc_q.bias | 256 |
| encoder.layers.0.self_attention.fc_k.weight | 65536 |
| encoder.layers.0.self_attention.fc_k.bias | 256 |
| encoder.layers.0.self_attention.fc_v.weight | 65536 |
| encoder.layers.0.self_attention.fc_v.bias | 256 |
| encoder.layers.0.self_attention.fc_o.weight | 65536 |
| encoder.layers.0.self_attention.fc_o.bias | 256 |
| encoder.layers.0.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.0.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.0.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.0.positionwise_feedforward.fc_2.bias | 256 |
| encoder.layers.1.self_attn_layer_norm.weight | 256 |
| encoder.layers.1.self_attn_layer_norm.bias | 256 |
| encoder.layers.1.ff_layer_norm.weight | 256 |
| encoder.layers.1.ff_layer_norm.bias | 256 |
| encoder.layers.1.self_attention.fc_q.weight | 65536 |
| encoder.layers.1.self_attention.fc_q.bias | 256 |
| encoder.layers.1.self_attention.fc_k.weight | 65536 |
| encoder.layers.1.self_attention.fc_k.bias | 256 |
| encoder.layers.1.self_attention.fc_v.weight | 65536 |
| encoder.layers.1.self_attention.fc_v.bias | 256 |
| encoder.layers.1.self_attention.fc_o.weight | 65536 |
| encoder.layers.1.self_attention.fc_o.bias | 256 |
| encoder.layers.1.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.1.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.1.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.1.positionwise_feedforward.fc_2.bias | 256 |
| encoder.layers.2.self_attn_layer_norm.weight | 256 |
| encoder.layers.2.self_attn_layer_norm.bias | 256 |
| encoder.layers.2.ff_layer_norm.weight | 256 |
| encoder.layers.2.ff_layer_norm.bias | 256 |
| encoder.layers.2.self_attention.fc_q.weight | 65536 |
| encoder.layers.2.self_attention.fc_q.bias | 256 |
| encoder.layers.2.self_attention.fc_k.weight | 65536 |
| encoder.layers.2.self_attention.fc_k.bias | 256 |
| encoder.layers.2.self_attention.fc_v.weight | 65536 |
| encoder.layers.2.self_attention.fc_v.bias | 256 |
| encoder.layers.2.self_attention.fc_o.weight | 65536 |
| encoder.layers.2.self_attention.fc_o.bias | 256 |
| encoder.layers.2.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.2.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.2.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.2.positionwise_feedforward.fc_2.bias | 256 |
| decoder.tok_embedding.weight | 3217664 |
| decoder.pos_embedding.weight | 25600 |
| decoder.layers.0.self_attn_layer_norm.weight | 256 |
| decoder.layers.0.self_attn_layer_norm.bias | 256 |
| decoder.layers.0.enc_attn_layer_norm.weight | 256 |
| decoder.layers.0.enc_attn_layer_norm.bias | 256 |
| decoder.layers.0.ff_layer_norm.weight | 256 |
| decoder.layers.0.ff_layer_norm.bias | 256 |
| decoder.layers.0.self_attention.fc_q.weight | 65536 |
| decoder.layers.0.self_attention.fc_q.bias | 256 |
| decoder.layers.0.self_attention.fc_k.weight | 65536 |
| decoder.layers.0.self_attention.fc_k.bias | 256 |
| decoder.layers.0.self_attention.fc_v.weight | 65536 |
| decoder.layers.0.self_attention.fc_v.bias | 256 |
| decoder.layers.0.self_attention.fc_o.weight | 65536 |
| decoder.layers.0.self_attention.fc_o.bias | 256 |
| decoder.layers.0.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_q.bias | 256 |
| decoder.layers.0.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_k.bias | 256 |
| decoder.layers.0.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_v.bias | 256 |
| decoder.layers.0.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_o.bias | 256 |
| decoder.layers.0.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.0.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.0.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.0.positionwise_feedforward.fc_2.bias | 256 |
| decoder.layers.1.self_attn_layer_norm.weight | 256 |
| decoder.layers.1.self_attn_layer_norm.bias | 256 |
| decoder.layers.1.enc_attn_layer_norm.weight | 256 |
| decoder.layers.1.enc_attn_layer_norm.bias | 256 |
| decoder.layers.1.ff_layer_norm.weight | 256 |
| decoder.layers.1.ff_layer_norm.bias | 256 |
| decoder.layers.1.self_attention.fc_q.weight | 65536 |
| decoder.layers.1.self_attention.fc_q.bias | 256 |
| decoder.layers.1.self_attention.fc_k.weight | 65536 |
| decoder.layers.1.self_attention.fc_k.bias | 256 |
| decoder.layers.1.self_attention.fc_v.weight | 65536 |
| decoder.layers.1.self_attention.fc_v.bias | 256 |
| decoder.layers.1.self_attention.fc_o.weight | 65536 |
| decoder.layers.1.self_attention.fc_o.bias | 256 |
| decoder.layers.1.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_q.bias | 256 |
| decoder.layers.1.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_k.bias | 256 |
| decoder.layers.1.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_v.bias | 256 |
| decoder.layers.1.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_o.bias | 256 |
| decoder.layers.1.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.1.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.1.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.1.positionwise_feedforward.fc_2.bias | 256 |
| decoder.layers.2.self_attn_layer_norm.weight | 256 |
| decoder.layers.2.self_attn_layer_norm.bias | 256 |
| decoder.layers.2.enc_attn_layer_norm.weight | 256 |
| decoder.layers.2.enc_attn_layer_norm.bias | 256 |
| decoder.layers.2.ff_layer_norm.weight | 256 |
| decoder.layers.2.ff_layer_norm.bias | 256 |
| decoder.layers.2.self_attention.fc_q.weight | 65536 |
| decoder.layers.2.self_attention.fc_q.bias | 256 |
| decoder.layers.2.self_attention.fc_k.weight | 65536 |
| decoder.layers.2.self_attention.fc_k.bias | 256 |
| decoder.layers.2.self_attention.fc_v.weight | 65536 |
| decoder.layers.2.self_attention.fc_v.bias | 256 |
| decoder.layers.2.self_attention.fc_o.weight | 65536 |
| decoder.layers.2.self_attention.fc_o.bias | 256 |
| decoder.layers.2.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_q.bias | 256 |
| decoder.layers.2.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_k.bias | 256 |
| decoder.layers.2.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_v.bias | 256 |
| decoder.layers.2.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_o.bias | 256 |
| decoder.layers.2.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.2.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.2.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.2.positionwise_feedforward.fc_2.bias | 256 |
| decoder.fc_out.weight | 3217664 |
| decoder.fc_out.bias | 12569 |
+-------------------------------------------------------+------------+
Total Trainable Params: 12506137
So, that's why I am unable to load the model. Because the model has a different parameter in local.
Even if I try to load the weights in local it gives me-
model.load_state_dict(torch.load(f"{model_name}.pt", map_location=device))
Error-
--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-24-f5baac4441a5> in <module>
----> 1 model.load_state_dict(torch.load(f"{model_name}_2.pt", map_location=device))
c:\anaconda\envs\lang_trans\lib\site-packages\torch\nn\modules\module.py in load_state_dict(self, state_dict, strict)
845 if len(error_msgs) > 0:
846 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 847 self.__class__.__name__, "\n\t".join(error_msgs)))
848 return _IncompatibleKeys(missing_keys, unexpected_keys)
849
RuntimeError: Error(s) in loading state_dict for Seq2Seq: size mismatch for decoder.tok_embedding.weight: copying a param with shape torch.Size([12538, 256]) from checkpoint, the shape in current model is torch.Size([12569, 256]). size mismatch for decoder.fc_out.weight: copying a param with shape torch.Size([12538, 256]) from checkpoint, the shape in current model is torch.Size([12569, 256]). size mismatch for decoder.fc_out.bias: copying a param with shape torch.Size([12538]) from checkpoint, the shape in current model is torch.Size([12569]).
The model param of the local must be wrong because in colab (device=CPU, runtime=None) I am able to load the weights after defining model class. But in the local machine the params changes, so I am unable to load the weights. I know it's weird, help me to find the solution.
You can check the full code of the model here-
<script src="https://gist.github.com/Dipeshpal/90c715a7b7f00845e20ef998bda35835.js"></script>
https://gist.github.com/Dipeshpal/90c715a7b7f00845e20ef998bda35835
After this model params change.
I am trying out neo4j to work on my grade project and I decided to create a database with the datagen provided by LDBC. This datagen came with these queries.
So I decided to run one of those queries and collected it's execution time (I ran them using cypher-shell command). So I collected these times:
553575, 558724, 556443, 556675, 551304, 555385, 552896
Note: I took the first run out because as said in this post the first time seemed to be a spike due to cold-cache.
Everything smooth until here, then I've decided to create some paths with new labels, making the database grow from 1.5GB to 2.4GB. The I decided to run again the queries expecting for them to be slower. For my surprise I saw that the times were substantially smaller.
382331, 380566, 405636, 405953, 407277, 391804, 371134, ....
From this post. I expected them to be like all be around a same time (like the first run) and perhaps bigger, but definitely not smaller.
For the purpose of this question, I am using an AWS machine, especifically the i3.4xlarge instance (16 vCPUs, 122GB RAM and 2x 1.9TB NVMe SSD). The current version I am using of Neo4j is 3.3.5 and I am using cypher-shell as stated above.
Additionally I have no other programs/process running on this machine so I think we can just eliminate that issue (and despite that being the case, it should slow them down, not speed them up).
Thanks in advance, I am just asking to see if someone may shed some light for me on this issue :)
EDIT:
The changes I've made were added some paths between :Message nodes and :Person's. Also some between :Person and :Person, some between :TagClass (other label) and :Person. Additionally, the DB had index on :Tag(name), :Person(name), :TagClass(name) and :Country(name) but for the purpose of executing the query, dropped those.
The original query:
PROFILE
MATCH (tag:Tag {name: 'Arnold_Schwarzenegger'})
MATCH (tag)<-[:HAS_TAG]-(message1:Message)-[:HAS_CREATOR]->(person1:Person)
MATCH (tag)<-[:HAS_TAG]-(message2:Message)-[:HAS_CREATOR]->(person1)
OPTIONAL MATCH (message2)<-[:LIKES]-(person2:Person)
OPTIONAL MATCH (person2)<-[:HAS_CREATOR]-(message3:Message)<-[like:LIKES]-(p3:Person)
RETURN
person1.id,
count(DISTINCT like) AS authorityScore
ORDER BY
authorityScore DESC,
person1.id ASC
LIMIT 100
These are profiles from both runs, due to space only added one plan. I can assure you, they're both the exact same (even on db hits, paths expanded, everything):
1st run:
| Plan | Statement | Version | Planner | Runtime | Time | DbHits | Rows |
| "PROFILE" | "READ_ONLY" | "CYPHER 3.3" | "COST" | "INTERPRETED" | 556443 | 0 | 100 |
2nd:
| Plan | Statement | Version | Planner | Runtime | Time | DbHits | Rows |
| "PROFILE" | "READ_ONLY" | "CYPHER 3.3" | "COST" | "INTERPRETED" | 371134 | 0 | 100 |
Plan:
+----------------------+----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| Operator | Estimated Rows | Rows | DB Hits | Cache H/M | Identifiers | Other |
+----------------------+----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +ProduceResults | 100 | 100 | 0 | 0/0 | anon[349], anon[355], authorityScore, person1.id | 0.0 |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +Projection | 100 | 100 | 0 | 0/0 | anon[349], anon[355], authorityScore, person1.id | 0.0; {person1.id : , authorityScore : } |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +Top | 100 | 100 | 0 | 0/0 | anon[349], anon[355] | 0.0; 100; anon[355], anon[349] |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +EagerAggregation | 126593 | 1842 | 49948155 | 0/0 | anon[349], anon[355] | 0.0; anon[349] |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +Apply | 16025884097 | 49948155 | 0 | 0/0 | person2, anon[219], message3, anon[60], tag, anon[136], p3, message1, anon[167], anon[271], message2, anon[91], person1, like | 0.0 |
| |\ +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Optional | 7317 | 49948155 | 0 | 0/0 | person2, message3, p3, anon[271], like | 0.0 |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Filter | 5037 | 49897917 | 49897917 | 0/0 | person2, message3, p3, anon[271], like | 0.0; p3:Person |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Expand(All) | 5037 | 49897917 | 108680609 | 0/0 | person2, message3, p3, anon[271], like | 0.0; (message3)<-[like:LIKES]-(p3) |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Filter | 7028 | 58782692 | 58782692 | 0/0 | anon[271], message3, person2 | 0.0; message3:Message |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Expand(All) | 7028 | 58782692 | 58866898 | 0/0 | anon[271], message3, person2 | 0.0; (person2)<-[anon[271]:HAS_CREATOR]-(message3) |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Argument | 7317 | 130898 | 0 | 0/0 | person2 | 0.0 |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +OptionalExpand(All) | 7317 | 130898 | 220191 | 0/0 | person2, anon[219], anon[60], tag, anon[136], message1, anon[167], message2, anon[91], person1 | 0.0; (message2)<-[anon[219]:LIKES]-(person2); person2:Person |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +NodeHashJoin | 7317 | 51779 | 0 | 0/0 | anon[60], tag, anon[136], message1, anon[167], message2, anon[91], person1 | 0.0; tag, person1 |
| |\ +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Filter | 341165 | 4241 | 4241 | 0/0 | tag, anon[136], anon[167], message2, person1 | 0.0; person1:Person |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Expand(All) | 341165 | 4241 | 8482 | 0/0 | tag, anon[136], anon[167], message2, person1 | 0.0; (message2)-[anon[167]:HAS_CREATOR]->(person1) |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Filter | 341165 | 4241 | 4646 | 0/0 | anon[136], message2, tag | 0.0; message2:Message |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Expand(All) | 372142 | 4646 | 4647 | 0/0 | anon[136], message2, tag | 0.0; (tag)<-[anon[136]:HAS_TAG]-(message2) |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +Filter | 1608 | 1 | 16080 | 0/0 | tag | 0.0; tag.name = { AUTOSTRING0} |
| | | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| | +NodeByLabelScan | 16080 | 16080 | 16081 | 0/0 | tag | 0.0; :Tag |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +Filter | 341165 | 4241 | 4241 | 0/0 | anon[60], tag, message1, anon[91], person1 | 0.0; person1:Person |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +Expand(All) | 341165 | 4241 | 8482 | 0/0 | anon[60], tag, message1, anon[91], person1 | 0.0; (message1)-[anon[91]:HAS_CREATOR]->(person1) |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +Filter | 341165 | 4241 | 4646 | 0/0 | anon[60], message1, tag | 0.0; message1:Message |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +Expand(All) | 372142 | 4646 | 4647 | 0/0 | anon[60], message1, tag | 0.0; (tag)<-[anon[60]:HAS_TAG]-(message1) |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +Filter | 1608 | 1 | 16080 | 0/0 | tag | 0.0; tag.name = { AUTOSTRING0} |
| | +----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------+
| +NodeByLabelScan | 16080 | 16080 | 16081 | 0/0 | tag | 0.0; :Tag |
+----------------------+----------------+----------+-----------+-----------+-------------------------------------------------------------------------------------------------------------------------------+------------------------------ --------------------------------+
... Results ...
100 rows available after 371134 ms, consumed after another 0 ms
Note: SO Doesn't allow me to write over 30000 characters so had to reduce the text.
The Cypher planner takes the current characteristics of the DB into account when deciding how to carry out a query. Any changes to a DB can alter the execution plan, which can greatly affect the timing.
The best way to understand timing differences after updating a DB is to do before-and-after comparisons of the query profile. Without knowing what query you are using, and especially without the before-and-after profile information (neither of which were provided), it is difficult to impossible to figure out why the timing changed.
The only guess that can be made, with the little information available, is that the timing had changed because the DB characteristics had changed in ways that caused the planner to alter the execution plan.
I have installed memsql 5.1.2 in following manner with following resources.
Google cloud server
HDD: 100GB
Machine type: n1-standard-4 (4 vCPUs, 15 GB memory)
Implementation:
2 MEMSQL NODES running on same machine on the following ports
3306 Master Aggregator
3307 Leaf
Resource Utilization:
Memory 14.16 GB / 14.69 GB
Paging 0 B/s
Database size - 10MB
1818 memsql 1.1% 77% /var/lib/memsql/leaf-3307/memsqld --defaults-file=/var/lib/memsql/leaf-3307/memsql.cnf --pid-file=/var/lib/memsql/leaf-3307/data/memsqld.pid --user=memsql
2736 memsql 0.3% 16% /var/lib/memsql/master-3306/memsqld --defaults-file=/var/lib/memsql/master-330
Note: There is no Swap memory implemented in the server.
Database size is taken by running a query on information_schema.TABLES.
All data resides as row store since we have to run queries by considering many relationships among tables.
As soon as the memsql is up the memory goes up to 70% and it keep on increasing and after 2-3 hours memsql gives the following error when try connect with it and connection also can not be done after that.
OperationalError: (1836, "Leaf 'xx.xxx.x.xx':3307 failed while executing this query. Try re-running the query.")
[Mon Mar 27 09:26:31.163455 2017] [:error] [pid 1718] [remote xxx.xxx.xxx.xxx:9956]
The only solution is to restart the server since it has taken up all the memory.
What I can do for this? Is there an issue in the way it's implemented? Any logs should I attach here?
Show status extended; query gives the following result
+-------------------------------------+------------------------------------------------------------------------+
| Variable_name | Value |
+-------------------------------------+------------------------------------------------------------------------+
| Aborted_clients | 48 |
| Aborted_connects | 1 |
| Bytes_received | 85962135 |
| Bytes_sent | 545322701 |
| Connections | 1626 |
| Max_used_connections | 69 |
| Queries | 364793 |
| Questions | 364793 |
| Threads_cached | 19 |
| Threads_connected | 50 |
| Threads_created | 69 |
| Threads_running | 1 |
| Threads_background | 1 |
| Threads_idle | 0 |
| Ready_queue | 0 |
| Idle_queue | 0 |
| Context_switches | 1626 |
| Context_switch_misses | 0 |
| Uptime | 22270 |
| Auto_attach_remaining_seconds | 0 |
| Data_directory | /var/lib/memsql/leaf-3307/data |
| Plancache_directory | /var/lib/memsql/leaf-3307/plancache |
| Transaction_logs_directory | /var/lib/memsql/leaf-3307/data/logs |
| Segments_directory | /var/lib/memsql/leaf-3307/data/columns |
| Snapshots_directory | /var/lib/memsql/leaf-3307/data/snapshots |
| Threads_waiting_for_disk_space | 0 |
| Seconds_until_expiration | -1 |
| License_key | 11111111111111111111111111111111 |
| License_type | community |
| Query_compilations | 62 |
| Query_compilation_failures | 0 |
| GCed_versions_last_sweep | 0 |
| Average_garbage_collection_duration | 21 ms |
| Total_server_memory | 9791.4 MB |
| Alloc_thread_stacks | 70.0 MB |
| Malloc_active_memory | 1254.7 (+0.0) MB |
| Malloc_cumulative_memory | 7315.5 (+0.2) MB |
| Buffer_manager_memory | 1787.8 MB |
| Buffer_manager_cached_memory | 77.2 (-0.1) MB |
| Buffer_manager_unrecycled_memory | 0.0 MB |
| Alloc_skiplist_tower | 263.8 MB |
| Alloc_variable | 501.4 MB |
| Alloc_large_variable | 2.4 MB |
| Alloc_table_primary | 752.6 MB |
| Alloc_deleted_version | 92.9 MB |
| Alloc_internal_key_node | 72.1 MB |
| Alloc_hash_buckets | 459.1 MB |
| Alloc_table_metadata_cache | 1.1 MB |
| Alloc_unit_images | 34.8 MB |
| Alloc_unit_ifn_thunks | 0.6 MB |
| Alloc_object_code_images | 11.6 MB |
| Alloc_compiled_unit_sections | 17.3 MB |
| Alloc_databases_list_entry | 17.9 MB |
| Alloc_plan_cache | 0.1 MB |
| Alloc_replication_large | 232.0 MB |
| Alloc_durability_large | 7239.1 MB |
| Alloc_sharding_partitions | 0.1 MB |
| Alloc_security | 0.1 MB |
| Alloc_log_replay | 0.9 MB |
| Alloc_client_connection | 3.0 MB |
| Alloc_protocol_packet | 6.1 (+0.1) MB |
| Alloc_large_incremental | 0.8 MB |
| Alloc_table_memory | 2144.2 MB |
| Alloc_variable_bucket_16 | allocs:10877846 alloc_MB:166.0 buffer_MB:179.0 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_24 | allocs:4275659 alloc_MB:97.9 buffer_MB:106.8 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_32 | allocs:2875801 alloc_MB:87.8 buffer_MB:93.4 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_40 | allocs:724489 alloc_MB:27.6 buffer_MB:31.0 cached_buffer_MB:1.2 |
| Alloc_variable_bucket_48 | allocs:377060 alloc_MB:17.3 buffer_MB:19.8 cached_buffer_MB:0.9 |
| Alloc_variable_bucket_56 | allocs:228720 alloc_MB:12.2 buffer_MB:14.0 cached_buffer_MB:0.8 |
| Alloc_variable_bucket_64 | allocs:150214 alloc_MB:9.2 buffer_MB:10.1 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_72 | allocs:35264 alloc_MB:2.4 buffer_MB:2.9 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_80 | allocs:14920 alloc_MB:1.1 buffer_MB:1.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_88 | allocs:5582 alloc_MB:0.5 buffer_MB:0.6 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_104 | allocs:8075 alloc_MB:0.8 buffer_MB:1.0 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_128 | allocs:8892 alloc_MB:1.1 buffer_MB:1.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_160 | allocs:17614 alloc_MB:2.7 buffer_MB:3.0 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_200 | allocs:30454 alloc_MB:5.8 buffer_MB:6.9 cached_buffer_MB:0.6 |
| Alloc_variable_bucket_248 | allocs:4875 alloc_MB:1.2 buffer_MB:1.5 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_312 | allocs:371 alloc_MB:0.1 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_384 | allocs:30 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_480 | allocs:11 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_600 | allocs:57 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_752 | allocs:62 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_936 | allocs:42 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1168 | allocs:106 alloc_MB:0.1 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1480 | allocs:126 alloc_MB:0.2 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1832 | allocs:0 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_2288 | allocs:1 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_2832 | allocs:33 alloc_MB:0.1 buffer_MB:1.1 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_3528 | allocs:16 alloc_MB:0.1 buffer_MB:0.5 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_4504 | allocs:49 alloc_MB:0.2 buffer_MB:0.8 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_5680 | allocs:66 alloc_MB:0.4 buffer_MB:1.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_6224 | allocs:30 alloc_MB:0.2 buffer_MB:1.0 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_7264 | allocs:94 alloc_MB:0.7 buffer_MB:1.5 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_9344 | allocs:70 alloc_MB:0.6 buffer_MB:2.6 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_11896 | allocs:14 alloc_MB:0.2 buffer_MB:2.4 cached_buffer_MB:1.2 |
| Alloc_variable_bucket_14544 | allocs:7 alloc_MB:0.1 buffer_MB:2.4 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_18696 | allocs:18 alloc_MB:0.3 buffer_MB:3.2 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_21816 | allocs:4 alloc_MB:0.1 buffer_MB:0.4 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_26184 | allocs:6 alloc_MB:0.1 buffer_MB:0.9 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_32728 | allocs:13 alloc_MB:0.4 buffer_MB:2.4 cached_buffer_MB:1.4 |
| Alloc_variable_bucket_43648 | allocs:12 alloc_MB:0.5 buffer_MB:1.4 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_65472 | allocs:7 alloc_MB:0.4 buffer_MB:2.8 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_130960 | allocs:3 alloc_MB:0.4 buffer_MB:2.2 cached_buffer_MB:1.9 |
| Alloc_variable_cached_buffers | 21.4 MB |
| Alloc_variable_allocated | 438.7 MB |
| Successful_read_queries | 9048 |
| Successful_write_queries | 19096 |
| Failed_read_queries | 0 |
| Failed_write_queries | 4 |
| Rows_returned_by_reads | 75939 |
| Rows_affected_by_writes | 245 |
| Execution_time_of_reads | 7864 ms |
| Execution_time_of_write | 180311 ms |
| Transaction_buffer_wait_time | 0 ms |
| Transaction_log_flush_wait_time | 0 ms |
| Row_lock_wait_time | 0 ms |
| Ssl_accept_renegotiates | 0 |
| Ssl_accepts | 0 |
| Ssl_callback_cache_hits | 0 |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_ctx_verify_depth | 18446744073709551615 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_default_timeout | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_finished_connects | 0 |
| Ssl_session_cache_hits | 0 |
| Ssl_session_cache_misses | 0 |
| Ssl_session_cache_overflows | 0 |
| Ssl_session_cache_size | 20480 |
| Ssl_session_cache_timeouts | 0 |
| Ssl_sessions_reused | 0 |
| Ssl_used_session_cache_entries | 0 |
| Ssl_verify_depth | 0 |
| Ssl_verify_mode | 0 |
| Ssl_cipher | |
| Ssl_cipher_list | |
| Ssl_version | |
| Ssl_session_cache_mode | SERVER |
+-------------------------------------+------------------------------------------------------------------------+
From the status output, we can see:
10GB total memory on the leaf node
7GB Alloc_durability_large
You can see what these variables mean here: https://help.memsql.com/hc/en-us/articles/115001091386-What-Is-Using-Memory-on-My-Leaves-
Most interesting is the large amount in Alloc_durability_large, which is unusual. Do you have a large number of databases and/or partitions? (You can check by counting the number of rows in SHOW DATABASES EXTENDED on the leaf nodoe.) Each will require a fixed amount of transaction buffer memory (default is 64 MB).
I am doing some search replace work using OLE Automation. The involved code is somehow the same I posted in This question.
On only one machine I have an "Invalid Variant Operation" error. This error appeared just after upgrading from MSOffice 2003 to 2010. But this occurs only on one pc, i tested on several pcs with office 2010 and the code works great.
I am really stuck, I tried to build with Eureka log And I will paste the call stack at the end of this question.
I hope somebody already stumbled into this problem so you can suggest.
The problem occured at a customer end, there are 100 machines connected to same DB and only on one it doesn't work.
Call Stack Information:
--------------------------------------------------------------------------------------------------
|Address |Module |Unit |Class |Procedure/Method |Line |
--------------------------------------------------------------------------------------------------
|*Exception Thread: ID=776; Priority=0; Class=; [Main] |
|------------------------------------------------------------------------------------------------|
|0043554F|Project1.exe|SysUtils.pas| |DoneExceptions |17031[25]|
|00435520|Project1.exe|SysUtils.pas| |DoneExceptions |17031[25]|
|00435CAA|Project1.exe|SysUtils.pas| |DoneExceptions |17031[25]|
|770F48F0|oleaut32.dll | | |VariantClear | |
|00435B43|Project1.exe|SysUtils.pas| |DoneExceptions |17031[25]|
|0043590C|Project1.exe|SysUtils.pas| |DoneExceptions |17031[25]|
|00435B48|Project1.exe|SysUtils.pas| |DoneExceptions |17031[25]|
|00435BE4|Project1.exe|SysUtils.pas| |DoneExceptions |17031[25]|
|00407316|Project1.exe|System.pas | |_HandleFinally |14330[39]|
|00407460|Project1.exe|System.pas | |_RaiseExcept |14712[1] |
|00635EED|Project1.exe|ComObj.pas | |DispCallError |2024[13] |
|004070DA|Project1.exe|System.pas | |_HandleAnyException |13728[81]|
|77E256A3|RPCRT4.dll | | |NdrClientCall2 | |
|7C80A174|kernel32.dll | | |WideCharToMultiByte | |
|0040C4DB|Project1.exe|System.pas | |LocaleCharsFromUnicode |29632[1] |
|0040C4C0|Project1.exe|System.pas | |LocaleCharsFromUnicode |29631[0] |
|00407EFC|Project1.exe|System.pas | |CharFromWChar |17375[3] |
|00407EDC|Project1.exe|System.pas | |CharFromWChar |17372[0] |
|00407F17|Project1.exe|System.pas | |CharFromWChar |17381[1] |
|00407F08|Project1.exe|System.pas | |CharFromWChar |17380[0] |
|004090CE|Project1.exe|System.pas | |_UStrToString |22054[8] |
|00404DA4|Project1.exe|System.pas | |Move |4808[1] |
|004090ED|Project1.exe|System.pas | |_UStrToString |22061[15]|
|7C920380|ntdll.dll | | |RtlImageNtHeader | |
|00407460|Project1.exe|System.pas | |_RaiseExcept |14712[1] |
|00635EED|Project1.exe|ComObj.pas | |DispCallError |2024[13] |
|004070DA|Project1.exe|System.pas | |_HandleAnyException |13728[81]|
|00407460|Project1.exe|System.pas | |_RaiseExcept |14712[1] |
|00635EED|Project1.exe|ComObj.pas | |DispCallError |2024[13] |
|7C913247|ntdll.dll | | |RtlConvertUlongToLargeInteger| |
|00404A95|Project1.exe|System.pas | |_ReallocMem |3871[22] |
|00404A88|Project1.exe|System.pas | |_ReallocMem |3850[1] |
|00409130|Project1.exe|System.pas | |_UStrSetLength |22152[27]|
|00408198|Project1.exe|System.pas | |_UStrAsg |17715[1] |
|0040C3EF|Project1.exe|System.pas | |UTF8ToUnicodeString |28576[11]|
|00407DB8|Project1.exe|System.pas | |_UStrClr |16954[1] |
|0040C404|Project1.exe|System.pas | |UTF8ToUnicodeString |28577[12]|
|7C91E485|ntdll.dll | | |KiUserApcDispatcher | |
|7C912B04|ntdll.dll | | |RtlInterlockedPushListSList | |
|77E943DB|SHLWAPI.dll | | |SHRegGetValueW | |
|77E943E8|SHLWAPI.dll | | |SHRegGetValueW | |
|00407460|Project1.exe|System.pas | |_RaiseExcept |14712[1] |
|00635EED|Project1.exe|ComObj.pas | |DispCallError |2024[13] |
|00635E48|Project1.exe|ComObj.pas | |DispCallError |2011[0] |
|00635F20|Project1.exe|ComObj.pas | |DispatchInvokeError |2031[1] |
|00635F1C|Project1.exe|ComObj.pas | |DispatchInvokeError |2031[1] |
|00635BF9|Project1.exe|ComObj.pas | |DispatchInvoke |1895[108]|
|00635940|Project1.exe|ComObj.pas | |DispatchInvoke |1787[0] |
|00635E3B|Project1.exe|ComObj.pas | |VarDispInvoke |2002[10] |
|774CF17D|ole32.dll | | |CoCreateInstanceEx | |
|77E25AE5|RPCRT4.dll | | |IUnknown_Release_Proxy | |
|00435C7A|Project1.exe|SysUtils.pas| |DoneExceptions |17031[25]|
|00435BE4|Project1.exe|SysUtils.pas| |DoneExceptions |17031[25]|
|004627DF|Project1.exe|Classes.pas |TThread |Create |11471[28]|
|004627D0|Project1.exe|Classes.pas |TThread |Create |11471[28]|
|00501FE5|Project1.exe|ActnList.pas|TContainedAction|Execute |448[8] |
|00501FB4|Project1.exe|ActnList.pas|TContainedAction|Execute |440[0] |
|00502DA0|Project1.exe|ActnList.pas|TCustomAction |Execute |1094[7] |
|00406498|Project1.exe|System.pas | |_CallDynaInst |11592[1] |
|004626A3|Project1.exe|Classes.pas |TThread |Create |11471[28]|
|00532DA4|Project1.exe|Menus.pas |TMenuItem |Click |2525[17] |
I use the following code to copy dbf/mdx files from one folder to another:
procedure TfrmMain.MyCopyFile(S1, S2: string);
begin
if not FileExists(S2) then
CopyFile(PCHAR(S1), PCHAR(S2), true)
else
if Application.MessageBox(PCHAR('Overwrite existing file ' + S2 + '?'), 'File exists in folder',MB_YESNO + MB_DEFBUTTON1) = IDYES
then CopyFile(PCHAR(S1), PCHAR(S2), false)
end;
The code works fine when table name stays the same.
If I change the name of the table:
MyCopyFile(CurPath + '\orders.dbf', NewPath + '\ordly.dbf');
MyCopyFile(CurPath + '\orders.mdx', NewPath + '\ordly.mdx');
When I try to open ordly.dbf I get an error message:
Corrupt table/index header.
File: C:\DATA\2011\ORDLY.MDX
the problem is due to the mdx format stores inside the name of the data file associated (table name). because that when you rename a mdxfile the index still points to the old name of data file.
check this link to see the structure of the mdx file.
The Structure of Multiple Index files (*.mdx)
0 | Version number *1| ^
|-----------------------| |
1 | Date of creation | |
2 | YYMMDD | |
3 | | |
|-----------------------| |
4 | Data file name | File
5 | (no extension) | Header
: : |
: : |
19 | | |
|-----------------------| |
20 | Block size | |
| | |
|-----------------------| |
22 | Block size adder N | |
| | |
|-----------------------| |
24 | Production index flag | |
|-----------------------| |
25 | No. of entries in tag | | *2
|-----------------------| |
26 | Length of tag | | *3
|-----------------------| |
27 | (Reserved) | |
|-----------------------| |
28 | No.of tags in use | |
| | |
|-----------------------| |
30 | (Reserved) | |
| | |
|-----------------------| |
32 | No.of pages in tagfile| |
| | |
| | |
35 | | |
|-----------------------| |
36 | Pointer to first free | |
| page | |
| | |
39 | | |
|-----------------------| |
40 | No.of block available | |
| | |
| | |
43 | | |
|-----------------------| |
44 | Date of last update | |
| YYMMDD | |
46 | | |
|-----------------------| |
47 | (Reserved) | |
|-----------------------| |
48 | (Garbage) | |
: : |
: : |
| | | ___|=======================|
543| | _V___ / 0 | Tag header page no. |
|-----------------------| | / | |
544| Tag table entries | Tag / | |
| | Table | 3 | |
:.......................: | | |-----------------------| Tag
: : | | 4 | Tag name | table
:.......................: | | : :
: : | / : :
: : | / | |
:.......................:__|_/ 14 | |
: : | |-----------------------|
: : | 15 | Key format *4 |
: : | |-----------------------|
:.......................:__|_ 16 | Forward tag thread (<)|
: : | \ |-----------------------|
: : | \ 17 | Forward tag thread (>)|
: : | \ |-----------------------|
: : | | 18 | Backward tag thread *5|
| | | | |-----------------------|
| | | | 19 | (Reserved) |
M*N| |__V__ | |-----------------------|
|=======================| ^ | 20 | Key type *6 |
0| Pointer to root page | | | |-----------------------|
| | | | 21 | (Reserved) |
| | | | : :
3| | | | : :
|-----------------------| | | 31 | |
4| File size in pages | Tag | |-----------------------|
| | header| 32 | (Garbage) |
| | | | : :
7| | | | | |
|-----------------------| | \ N | |
8| Key format *7 | | \____|=======================|
|-----------------------| |
9| Key type *8 | |
|-----------------------| |
10| (Reserved) | |
| | |
|-----------------------| |
12| Index key length *9 | |
| | |
|-----------------------| |
14| Max.no.of keys/page | |
| | |
|-----------------------| |
16| Secondary key type *10| |
| | |
|-----------------------| |
18| Index key item length | |
| | |
|-----------------------| |
20| (Reserved) | |
| | |
| | |
|-----------------------| |
23| Unique flag | |
|-----------------------| |
| | |
: : |
: :__V__
N*M|=======================|