I have created the model and save the weights using google colab. Now I have created a prediction script.
The prediction script contains the model class. I am trying to load the model weights using the following method-
Saving & Loading Model Across Devices
Save on GPU, Load on CPU
Save:
torch.save(model.state_dict(), PATH)
Load:
device = torch.device('cpu')
model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH, map_location=device))
The above method should work, right? Yes.
But when I am trying to do so I have different parameters of the model in Google Colab (Prediction, runtime-None, device=CPU) and different in my local machine (prediction, device=cpu)
Model Params in Colab-
def count_parameters(model):
return sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f'The model has {count_parameters(model):,} trainable parameters')
The model has 12,490,234 trainable parameters
+-------------------------------------------------------+------------+
| Modules | Parameters |
+-------------------------------------------------------+------------+
| encoder.tok_embedding.weight | 2053376 |
| encoder.pos_embedding.weight | 25600 |
| encoder.layers.0.self_attn_layer_norm.weight | 256 |
| encoder.layers.0.self_attn_layer_norm.bias | 256 |
| encoder.layers.0.ff_layer_norm.weight | 256 |
| encoder.layers.0.ff_layer_norm.bias | 256 |
| encoder.layers.0.self_attention.fc_q.weight | 65536 |
| encoder.layers.0.self_attention.fc_q.bias | 256 |
| encoder.layers.0.self_attention.fc_k.weight | 65536 |
| encoder.layers.0.self_attention.fc_k.bias | 256 |
| encoder.layers.0.self_attention.fc_v.weight | 65536 |
| encoder.layers.0.self_attention.fc_v.bias | 256 |
| encoder.layers.0.self_attention.fc_o.weight | 65536 |
| encoder.layers.0.self_attention.fc_o.bias | 256 |
| encoder.layers.0.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.0.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.0.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.0.positionwise_feedforward.fc_2.bias | 256 |
| encoder.layers.1.self_attn_layer_norm.weight | 256 |
| encoder.layers.1.self_attn_layer_norm.bias | 256 |
| encoder.layers.1.ff_layer_norm.weight | 256 |
| encoder.layers.1.ff_layer_norm.bias | 256 |
| encoder.layers.1.self_attention.fc_q.weight | 65536 |
| encoder.layers.1.self_attention.fc_q.bias | 256 |
| encoder.layers.1.self_attention.fc_k.weight | 65536 |
| encoder.layers.1.self_attention.fc_k.bias | 256 |
| encoder.layers.1.self_attention.fc_v.weight | 65536 |
| encoder.layers.1.self_attention.fc_v.bias | 256 |
| encoder.layers.1.self_attention.fc_o.weight | 65536 |
| encoder.layers.1.self_attention.fc_o.bias | 256 |
| encoder.layers.1.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.1.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.1.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.1.positionwise_feedforward.fc_2.bias | 256 |
| encoder.layers.2.self_attn_layer_norm.weight | 256 |
| encoder.layers.2.self_attn_layer_norm.bias | 256 |
| encoder.layers.2.ff_layer_norm.weight | 256 |
| encoder.layers.2.ff_layer_norm.bias | 256 |
| encoder.layers.2.self_attention.fc_q.weight | 65536 |
| encoder.layers.2.self_attention.fc_q.bias | 256 |
| encoder.layers.2.self_attention.fc_k.weight | 65536 |
| encoder.layers.2.self_attention.fc_k.bias | 256 |
| encoder.layers.2.self_attention.fc_v.weight | 65536 |
| encoder.layers.2.self_attention.fc_v.bias | 256 |
| encoder.layers.2.self_attention.fc_o.weight | 65536 |
| encoder.layers.2.self_attention.fc_o.bias | 256 |
| encoder.layers.2.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.2.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.2.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.2.positionwise_feedforward.fc_2.bias | 256 |
| decoder.tok_embedding.weight | 3209728 |
| decoder.pos_embedding.weight | 25600 |
| decoder.layers.0.self_attn_layer_norm.weight | 256 |
| decoder.layers.0.self_attn_layer_norm.bias | 256 |
| decoder.layers.0.enc_attn_layer_norm.weight | 256 |
| decoder.layers.0.enc_attn_layer_norm.bias | 256 |
| decoder.layers.0.ff_layer_norm.weight | 256 |
| decoder.layers.0.ff_layer_norm.bias | 256 |
| decoder.layers.0.self_attention.fc_q.weight | 65536 |
| decoder.layers.0.self_attention.fc_q.bias | 256 |
| decoder.layers.0.self_attention.fc_k.weight | 65536 |
| decoder.layers.0.self_attention.fc_k.bias | 256 |
| decoder.layers.0.self_attention.fc_v.weight | 65536 |
| decoder.layers.0.self_attention.fc_v.bias | 256 |
| decoder.layers.0.self_attention.fc_o.weight | 65536 |
| decoder.layers.0.self_attention.fc_o.bias | 256 |
| decoder.layers.0.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_q.bias | 256 |
| decoder.layers.0.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_k.bias | 256 |
| decoder.layers.0.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_v.bias | 256 |
| decoder.layers.0.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_o.bias | 256 |
| decoder.layers.0.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.0.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.0.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.0.positionwise_feedforward.fc_2.bias | 256 |
| decoder.layers.1.self_attn_layer_norm.weight | 256 |
| decoder.layers.1.self_attn_layer_norm.bias | 256 |
| decoder.layers.1.enc_attn_layer_norm.weight | 256 |
| decoder.layers.1.enc_attn_layer_norm.bias | 256 |
| decoder.layers.1.ff_layer_norm.weight | 256 |
| decoder.layers.1.ff_layer_norm.bias | 256 |
| decoder.layers.1.self_attention.fc_q.weight | 65536 |
| decoder.layers.1.self_attention.fc_q.bias | 256 |
| decoder.layers.1.self_attention.fc_k.weight | 65536 |
| decoder.layers.1.self_attention.fc_k.bias | 256 |
| decoder.layers.1.self_attention.fc_v.weight | 65536 |
| decoder.layers.1.self_attention.fc_v.bias | 256 |
| decoder.layers.1.self_attention.fc_o.weight | 65536 |
| decoder.layers.1.self_attention.fc_o.bias | 256 |
| decoder.layers.1.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_q.bias | 256 |
| decoder.layers.1.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_k.bias | 256 |
| decoder.layers.1.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_v.bias | 256 |
| decoder.layers.1.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_o.bias | 256 |
| decoder.layers.1.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.1.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.1.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.1.positionwise_feedforward.fc_2.bias | 256 |
| decoder.layers.2.self_attn_layer_norm.weight | 256 |
| decoder.layers.2.self_attn_layer_norm.bias | 256 |
| decoder.layers.2.enc_attn_layer_norm.weight | 256 |
| decoder.layers.2.enc_attn_layer_norm.bias | 256 |
| decoder.layers.2.ff_layer_norm.weight | 256 |
| decoder.layers.2.ff_layer_norm.bias | 256 |
| decoder.layers.2.self_attention.fc_q.weight | 65536 |
| decoder.layers.2.self_attention.fc_q.bias | 256 |
| decoder.layers.2.self_attention.fc_k.weight | 65536 |
| decoder.layers.2.self_attention.fc_k.bias | 256 |
| decoder.layers.2.self_attention.fc_v.weight | 65536 |
| decoder.layers.2.self_attention.fc_v.bias | 256 |
| decoder.layers.2.self_attention.fc_o.weight | 65536 |
| decoder.layers.2.self_attention.fc_o.bias | 256 |
| decoder.layers.2.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_q.bias | 256 |
| decoder.layers.2.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_k.bias | 256 |
| decoder.layers.2.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_v.bias | 256 |
| decoder.layers.2.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_o.bias | 256 |
| decoder.layers.2.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.2.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.2.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.2.positionwise_feedforward.fc_2.bias | 256 |
| decoder.fc_out.weight | 3209728 |
| decoder.fc_out.bias | 12538 |
+-------------------------------------------------------+------------+
Total Trainable Params: 12490234
Model Params in Local-
def count_parameters(model):
return sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f'The model has {count_parameters(model):,} trainable parameters')
The model has 12,506,137 trainable parameters
+-------------------------------------------------------+------------+
| Modules | Parameters |
+-------------------------------------------------------+------------+
| encoder.tok_embedding.weight | 2053376 |
| encoder.pos_embedding.weight | 25600 |
| encoder.layers.0.self_attn_layer_norm.weight | 256 |
| encoder.layers.0.self_attn_layer_norm.bias | 256 |
| encoder.layers.0.ff_layer_norm.weight | 256 |
| encoder.layers.0.ff_layer_norm.bias | 256 |
| encoder.layers.0.self_attention.fc_q.weight | 65536 |
| encoder.layers.0.self_attention.fc_q.bias | 256 |
| encoder.layers.0.self_attention.fc_k.weight | 65536 |
| encoder.layers.0.self_attention.fc_k.bias | 256 |
| encoder.layers.0.self_attention.fc_v.weight | 65536 |
| encoder.layers.0.self_attention.fc_v.bias | 256 |
| encoder.layers.0.self_attention.fc_o.weight | 65536 |
| encoder.layers.0.self_attention.fc_o.bias | 256 |
| encoder.layers.0.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.0.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.0.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.0.positionwise_feedforward.fc_2.bias | 256 |
| encoder.layers.1.self_attn_layer_norm.weight | 256 |
| encoder.layers.1.self_attn_layer_norm.bias | 256 |
| encoder.layers.1.ff_layer_norm.weight | 256 |
| encoder.layers.1.ff_layer_norm.bias | 256 |
| encoder.layers.1.self_attention.fc_q.weight | 65536 |
| encoder.layers.1.self_attention.fc_q.bias | 256 |
| encoder.layers.1.self_attention.fc_k.weight | 65536 |
| encoder.layers.1.self_attention.fc_k.bias | 256 |
| encoder.layers.1.self_attention.fc_v.weight | 65536 |
| encoder.layers.1.self_attention.fc_v.bias | 256 |
| encoder.layers.1.self_attention.fc_o.weight | 65536 |
| encoder.layers.1.self_attention.fc_o.bias | 256 |
| encoder.layers.1.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.1.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.1.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.1.positionwise_feedforward.fc_2.bias | 256 |
| encoder.layers.2.self_attn_layer_norm.weight | 256 |
| encoder.layers.2.self_attn_layer_norm.bias | 256 |
| encoder.layers.2.ff_layer_norm.weight | 256 |
| encoder.layers.2.ff_layer_norm.bias | 256 |
| encoder.layers.2.self_attention.fc_q.weight | 65536 |
| encoder.layers.2.self_attention.fc_q.bias | 256 |
| encoder.layers.2.self_attention.fc_k.weight | 65536 |
| encoder.layers.2.self_attention.fc_k.bias | 256 |
| encoder.layers.2.self_attention.fc_v.weight | 65536 |
| encoder.layers.2.self_attention.fc_v.bias | 256 |
| encoder.layers.2.self_attention.fc_o.weight | 65536 |
| encoder.layers.2.self_attention.fc_o.bias | 256 |
| encoder.layers.2.positionwise_feedforward.fc_1.weight | 131072 |
| encoder.layers.2.positionwise_feedforward.fc_1.bias | 512 |
| encoder.layers.2.positionwise_feedforward.fc_2.weight | 131072 |
| encoder.layers.2.positionwise_feedforward.fc_2.bias | 256 |
| decoder.tok_embedding.weight | 3217664 |
| decoder.pos_embedding.weight | 25600 |
| decoder.layers.0.self_attn_layer_norm.weight | 256 |
| decoder.layers.0.self_attn_layer_norm.bias | 256 |
| decoder.layers.0.enc_attn_layer_norm.weight | 256 |
| decoder.layers.0.enc_attn_layer_norm.bias | 256 |
| decoder.layers.0.ff_layer_norm.weight | 256 |
| decoder.layers.0.ff_layer_norm.bias | 256 |
| decoder.layers.0.self_attention.fc_q.weight | 65536 |
| decoder.layers.0.self_attention.fc_q.bias | 256 |
| decoder.layers.0.self_attention.fc_k.weight | 65536 |
| decoder.layers.0.self_attention.fc_k.bias | 256 |
| decoder.layers.0.self_attention.fc_v.weight | 65536 |
| decoder.layers.0.self_attention.fc_v.bias | 256 |
| decoder.layers.0.self_attention.fc_o.weight | 65536 |
| decoder.layers.0.self_attention.fc_o.bias | 256 |
| decoder.layers.0.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_q.bias | 256 |
| decoder.layers.0.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_k.bias | 256 |
| decoder.layers.0.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_v.bias | 256 |
| decoder.layers.0.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.0.encoder_attention.fc_o.bias | 256 |
| decoder.layers.0.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.0.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.0.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.0.positionwise_feedforward.fc_2.bias | 256 |
| decoder.layers.1.self_attn_layer_norm.weight | 256 |
| decoder.layers.1.self_attn_layer_norm.bias | 256 |
| decoder.layers.1.enc_attn_layer_norm.weight | 256 |
| decoder.layers.1.enc_attn_layer_norm.bias | 256 |
| decoder.layers.1.ff_layer_norm.weight | 256 |
| decoder.layers.1.ff_layer_norm.bias | 256 |
| decoder.layers.1.self_attention.fc_q.weight | 65536 |
| decoder.layers.1.self_attention.fc_q.bias | 256 |
| decoder.layers.1.self_attention.fc_k.weight | 65536 |
| decoder.layers.1.self_attention.fc_k.bias | 256 |
| decoder.layers.1.self_attention.fc_v.weight | 65536 |
| decoder.layers.1.self_attention.fc_v.bias | 256 |
| decoder.layers.1.self_attention.fc_o.weight | 65536 |
| decoder.layers.1.self_attention.fc_o.bias | 256 |
| decoder.layers.1.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_q.bias | 256 |
| decoder.layers.1.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_k.bias | 256 |
| decoder.layers.1.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_v.bias | 256 |
| decoder.layers.1.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.1.encoder_attention.fc_o.bias | 256 |
| decoder.layers.1.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.1.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.1.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.1.positionwise_feedforward.fc_2.bias | 256 |
| decoder.layers.2.self_attn_layer_norm.weight | 256 |
| decoder.layers.2.self_attn_layer_norm.bias | 256 |
| decoder.layers.2.enc_attn_layer_norm.weight | 256 |
| decoder.layers.2.enc_attn_layer_norm.bias | 256 |
| decoder.layers.2.ff_layer_norm.weight | 256 |
| decoder.layers.2.ff_layer_norm.bias | 256 |
| decoder.layers.2.self_attention.fc_q.weight | 65536 |
| decoder.layers.2.self_attention.fc_q.bias | 256 |
| decoder.layers.2.self_attention.fc_k.weight | 65536 |
| decoder.layers.2.self_attention.fc_k.bias | 256 |
| decoder.layers.2.self_attention.fc_v.weight | 65536 |
| decoder.layers.2.self_attention.fc_v.bias | 256 |
| decoder.layers.2.self_attention.fc_o.weight | 65536 |
| decoder.layers.2.self_attention.fc_o.bias | 256 |
| decoder.layers.2.encoder_attention.fc_q.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_q.bias | 256 |
| decoder.layers.2.encoder_attention.fc_k.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_k.bias | 256 |
| decoder.layers.2.encoder_attention.fc_v.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_v.bias | 256 |
| decoder.layers.2.encoder_attention.fc_o.weight | 65536 |
| decoder.layers.2.encoder_attention.fc_o.bias | 256 |
| decoder.layers.2.positionwise_feedforward.fc_1.weight | 131072 |
| decoder.layers.2.positionwise_feedforward.fc_1.bias | 512 |
| decoder.layers.2.positionwise_feedforward.fc_2.weight | 131072 |
| decoder.layers.2.positionwise_feedforward.fc_2.bias | 256 |
| decoder.fc_out.weight | 3217664 |
| decoder.fc_out.bias | 12569 |
+-------------------------------------------------------+------------+
Total Trainable Params: 12506137
So, that's why I am unable to load the model. Because the model has a different parameter in local.
Even if I try to load the weights in local it gives me-
model.load_state_dict(torch.load(f"{model_name}.pt", map_location=device))
Error-
--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-24-f5baac4441a5> in <module>
----> 1 model.load_state_dict(torch.load(f"{model_name}_2.pt", map_location=device))
c:\anaconda\envs\lang_trans\lib\site-packages\torch\nn\modules\module.py in load_state_dict(self, state_dict, strict)
845 if len(error_msgs) > 0:
846 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 847 self.__class__.__name__, "\n\t".join(error_msgs)))
848 return _IncompatibleKeys(missing_keys, unexpected_keys)
849
RuntimeError: Error(s) in loading state_dict for Seq2Seq: size mismatch for decoder.tok_embedding.weight: copying a param with shape torch.Size([12538, 256]) from checkpoint, the shape in current model is torch.Size([12569, 256]). size mismatch for decoder.fc_out.weight: copying a param with shape torch.Size([12538, 256]) from checkpoint, the shape in current model is torch.Size([12569, 256]). size mismatch for decoder.fc_out.bias: copying a param with shape torch.Size([12538]) from checkpoint, the shape in current model is torch.Size([12569]).
The model param of the local must be wrong because in colab (device=CPU, runtime=None) I am able to load the weights after defining model class. But in the local machine the params changes, so I am unable to load the weights. I know it's weird, help me to find the solution.
You can check the full code of the model here-
<script src="https://gist.github.com/Dipeshpal/90c715a7b7f00845e20ef998bda35835.js"></script>
https://gist.github.com/Dipeshpal/90c715a7b7f00845e20ef998bda35835
After this model params change.
I create a one master stream with value as array. I would get array fields at one and stream it to ne ksql stream.
At first i create the master stream:
CREATE STREAM runtime_master_stream
(timestamp BIGINT,
opcuaObject VARCHAR,
value array<DOUBLE>)
WITH (KAFKA_TOPIC='runtime_master', VALUE_FORMAT='JSON');
Output is:
{"ROWTIME":1557317077577,"ROWKEY":"\u0000\u0000\u0001j�T�I","timestamp":1557317069589,"opcuaObject":"DatBetrZ.BetrZStdOM","value":[19.737154,512.0,320.18024,423.87027,399.99384,292.1198,450.821]}
Then i create new stream to get the arrayfields at one:
CREATE STREAM runtime_std_om_all_knife_stream
WITH (TIMESTAMP='timestamp',
PARTITIONS=4,
VALUE_FORMAT='JSON') AS
SELECT
timestamp,
opcuaObject,
value[0] AS knife1,
value[1] AS knife02,
value[2] AS knife03,
value[3] AS knife04,
value[4] AS knife05,
value[5] AS knife05,
value[6] AS knife06
FROM RUNTIME_MASTER_STREAM
WHERE opcuaObject ='DatBetrZ.BetrZStdOM';
Output:
{"ROWTIME":1557317170312,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"DatBetrZ.BetrZStdOM","KNIFE1":19.737154,"KNIFE02":512.0,"KNIFE03":320.18024,"KNIFE04":42
3.87027,"KNIFE05":400.02216,"KNIFE06":292.1198,}
What i need is this output in one ksql stream EXAMPLE:
{"ROWTIME":1557317170312,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife1","VALUE":19.737154}
{"ROWTIME":1557317170313,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife02","VALUE":19.737154}
{"ROWTIME":1557317170312,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife03","VALUE":19.737154}
{"ROWTIME":1557317170313,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife04","VALUE":19.737154}
{"ROWTIME":1557317170312,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife05","VALUE":19.737154}
{"ROWTIME":1557317170313,"ROWKEY":"\u0000\u0000\u0001j�V4�","TIMESTAMP":1557317162337,"OPCUAOBJECT":"Knife06","VALUE":39.737154}
Any idea how to get this output with one stream in kafka ksql?
What you're looking for is an EXPLODE/UNNEST function, which is not currently available but for which there is an existing issue that you can upvote if you think it would be useful.
A workaround for you could be something like this, in which you brute-force populate the target stream with INSERT INTO iterating over all the possible indices of your array:
Populate some test data into a Kafka topic
$ curl "https://api.mockaroo.com/api/440970e0?count=5&key=ff7856d0" | \
kafkacat -P -b localhost -t car_data_01
Examine the data in KSQL:
ksql> PRINT 'car_data_01' FROM BEGINNING;
Format:JSON
{"ROWTIME":1557392065409,"ROWKEY":"null","timestamp":"1533200557","car":"Oldsmobile","value":[68.93,53.58]}
{"ROWTIME":1557392065409,"ROWKEY":"null","timestamp":"1548442477","car":"Mercury","value":[60.09,69.07,63.77,63.13]}
{"ROWTIME":1557392065409,"ROWKEY":"null","timestamp":"1544928225","car":"Volkswagen","value":[59.77,6.94,97.7,30.86,16.9]}
{"ROWTIME":1557392065409,"ROWKEY":"null","timestamp":"1545383393","car":"Nissan","value":[13.32]}
{"ROWTIME":1557392065412,"ROWKEY":"null","timestamp":"1552825010","car":"Hyundai","value":[12.92]}
Create a stream over the data:
CREATE STREAM CAR_DATA (timestamp BIGINT, CAR VARCHAR, VALUE ARRAY<DOUBLE>) WITH (KAFKA_TOPIC='car_data_01', VALUE_FORMAT='JSON');
ksql> SELECT TIMESTAMP, CAR, VALUE[0], VALUE[1], VALUE[2] FROM CAR_DATA;
1533200557 | Oldsmobile | 68.93 | 53.58 | null
1548442477 | Mercury | 60.09 | 69.07 | 63.77
1544928225 | Volkswagen | 59.77 | 6.94 | 97.7
1545383393 | Nissan | 13.32 | null | null
1552825010 | Hyundai | 12.92 | null | null
Create the output stream, to start with just containing the zero-index elements of the array:
CREATE STREAM CAR_DATA_EXPLODED AS SELECT TIMESTAMP, CAR, 'Sensor 00' AS SOURCE, VALUE[0] AS VALUE FROM CAR_DATA WHERE VALUE[0] IS NOT NULL;
ksql> SELECT * FROM CAR_DATA_EXPLODED;
1557392065409 | null | 1544928225 | Volkswagen | Sensor 00 | 59.77
1557392065409 | null | 1545383393 | Nissan | Sensor 00 | 13.32
1557392065409 | null | 1533200557 | Oldsmobile | Sensor 00 | 68.93
1557392065412 | null | 1552825010 | Hyundai | Sensor 00 | 12.92
1557392065409 | null | 1548442477 | Mercury | Sensor 00 | 60.09
Insert the remaining array indices to the new stream:
CREATE STREAM CAR_DATA_EXPLODED_00 AS SELECT TIMESTAMP, CAR, 'Sensor 00' AS SOURCE, VALUE[0] AS VALUE FROM CAR_DATA
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 01' AS SOURCE, VALUE[1] AS VALUE FROM CAR_DATA WHERE VALUE[1] IS NOT NULL;
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 02' AS SOURCE, VALUE[2] AS VALUE FROM CAR_DATA WHERE VALUE[2] IS NOT NULL;
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 03' AS SOURCE, VALUE[3] AS VALUE FROM CAR_DATA WHERE VALUE[3] IS NOT NULL;
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 04' AS SOURCE, VALUE[4] AS VALUE FROM CAR_DATA WHERE VALUE[4] IS NOT NULL;
INSERT INTO CAR_DATA_EXPLODED_00 SELECT TIMESTAMP, CAR, 'Sensor 05' AS SOURCE, VALUE[5] AS VALUE FROM CAR_DATA WHERE VALUE[5] IS NOT NULL;
Examine the exploded data:
ksql> SELECT * FROM CAR_DATA_EXPLODED_00;
1557392065409 | null | 1533200557 | Oldsmobile | Sensor 00 | 68.93
1557392065409 | null | 1545383393 | Nissan | Sensor 00 | 13.32
1557392065409 | null | 1544928225 | Volkswagen | Sensor 00 | 59.77
1557392065409 | null | 1548442477 | Mercury | Sensor 02 | 63.77
1557392065409 | null | 1544928225 | Volkswagen | Sensor 03 | 30.86
1557392065409 | null | 1544928225 | Volkswagen | Sensor 04 | 16.9
1557392065409 | null | 1533200557 | Oldsmobile | Sensor 01 | 53.58
1557392065409 | null | 1544928225 | Volkswagen | Sensor 02 | 97.7
1557392065412 | null | 1552825010 | Hyundai | Sensor 00 | 12.92
1557392065409 | null | 1548442477 | Mercury | Sensor 01 | 69.07
1557392065409 | null | 1548442477 | Mercury | Sensor 00 | 60.09
1557392065409 | null | 1544928225 | Volkswagen | Sensor 01 | 6.94
1557392065409 | null | 1548442477 | Mercury | Sensor 03 | 63.13
ksql> SELECT * FROM CAR_DATA_EXPLODED_00 WHERE SOURCE='Sensor 03';
1557392065409 | null | 1544928225 | Volkswagen | Sensor 03 | 30.86
1557392065409 | null | 1548442477 | Mercury | Sensor 03 | 63.13
I have installed memsql 5.1.2 in following manner with following resources.
Google cloud server
HDD: 100GB
Machine type: n1-standard-4 (4 vCPUs, 15 GB memory)
Implementation:
2 MEMSQL NODES running on same machine on the following ports
3306 Master Aggregator
3307 Leaf
Resource Utilization:
Memory 14.16 GB / 14.69 GB
Paging 0 B/s
Database size - 10MB
1818 memsql 1.1% 77% /var/lib/memsql/leaf-3307/memsqld --defaults-file=/var/lib/memsql/leaf-3307/memsql.cnf --pid-file=/var/lib/memsql/leaf-3307/data/memsqld.pid --user=memsql
2736 memsql 0.3% 16% /var/lib/memsql/master-3306/memsqld --defaults-file=/var/lib/memsql/master-330
Note: There is no Swap memory implemented in the server.
Database size is taken by running a query on information_schema.TABLES.
All data resides as row store since we have to run queries by considering many relationships among tables.
As soon as the memsql is up the memory goes up to 70% and it keep on increasing and after 2-3 hours memsql gives the following error when try connect with it and connection also can not be done after that.
OperationalError: (1836, "Leaf 'xx.xxx.x.xx':3307 failed while executing this query. Try re-running the query.")
[Mon Mar 27 09:26:31.163455 2017] [:error] [pid 1718] [remote xxx.xxx.xxx.xxx:9956]
The only solution is to restart the server since it has taken up all the memory.
What I can do for this? Is there an issue in the way it's implemented? Any logs should I attach here?
Show status extended; query gives the following result
+-------------------------------------+------------------------------------------------------------------------+
| Variable_name | Value |
+-------------------------------------+------------------------------------------------------------------------+
| Aborted_clients | 48 |
| Aborted_connects | 1 |
| Bytes_received | 85962135 |
| Bytes_sent | 545322701 |
| Connections | 1626 |
| Max_used_connections | 69 |
| Queries | 364793 |
| Questions | 364793 |
| Threads_cached | 19 |
| Threads_connected | 50 |
| Threads_created | 69 |
| Threads_running | 1 |
| Threads_background | 1 |
| Threads_idle | 0 |
| Ready_queue | 0 |
| Idle_queue | 0 |
| Context_switches | 1626 |
| Context_switch_misses | 0 |
| Uptime | 22270 |
| Auto_attach_remaining_seconds | 0 |
| Data_directory | /var/lib/memsql/leaf-3307/data |
| Plancache_directory | /var/lib/memsql/leaf-3307/plancache |
| Transaction_logs_directory | /var/lib/memsql/leaf-3307/data/logs |
| Segments_directory | /var/lib/memsql/leaf-3307/data/columns |
| Snapshots_directory | /var/lib/memsql/leaf-3307/data/snapshots |
| Threads_waiting_for_disk_space | 0 |
| Seconds_until_expiration | -1 |
| License_key | 11111111111111111111111111111111 |
| License_type | community |
| Query_compilations | 62 |
| Query_compilation_failures | 0 |
| GCed_versions_last_sweep | 0 |
| Average_garbage_collection_duration | 21 ms |
| Total_server_memory | 9791.4 MB |
| Alloc_thread_stacks | 70.0 MB |
| Malloc_active_memory | 1254.7 (+0.0) MB |
| Malloc_cumulative_memory | 7315.5 (+0.2) MB |
| Buffer_manager_memory | 1787.8 MB |
| Buffer_manager_cached_memory | 77.2 (-0.1) MB |
| Buffer_manager_unrecycled_memory | 0.0 MB |
| Alloc_skiplist_tower | 263.8 MB |
| Alloc_variable | 501.4 MB |
| Alloc_large_variable | 2.4 MB |
| Alloc_table_primary | 752.6 MB |
| Alloc_deleted_version | 92.9 MB |
| Alloc_internal_key_node | 72.1 MB |
| Alloc_hash_buckets | 459.1 MB |
| Alloc_table_metadata_cache | 1.1 MB |
| Alloc_unit_images | 34.8 MB |
| Alloc_unit_ifn_thunks | 0.6 MB |
| Alloc_object_code_images | 11.6 MB |
| Alloc_compiled_unit_sections | 17.3 MB |
| Alloc_databases_list_entry | 17.9 MB |
| Alloc_plan_cache | 0.1 MB |
| Alloc_replication_large | 232.0 MB |
| Alloc_durability_large | 7239.1 MB |
| Alloc_sharding_partitions | 0.1 MB |
| Alloc_security | 0.1 MB |
| Alloc_log_replay | 0.9 MB |
| Alloc_client_connection | 3.0 MB |
| Alloc_protocol_packet | 6.1 (+0.1) MB |
| Alloc_large_incremental | 0.8 MB |
| Alloc_table_memory | 2144.2 MB |
| Alloc_variable_bucket_16 | allocs:10877846 alloc_MB:166.0 buffer_MB:179.0 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_24 | allocs:4275659 alloc_MB:97.9 buffer_MB:106.8 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_32 | allocs:2875801 alloc_MB:87.8 buffer_MB:93.4 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_40 | allocs:724489 alloc_MB:27.6 buffer_MB:31.0 cached_buffer_MB:1.2 |
| Alloc_variable_bucket_48 | allocs:377060 alloc_MB:17.3 buffer_MB:19.8 cached_buffer_MB:0.9 |
| Alloc_variable_bucket_56 | allocs:228720 alloc_MB:12.2 buffer_MB:14.0 cached_buffer_MB:0.8 |
| Alloc_variable_bucket_64 | allocs:150214 alloc_MB:9.2 buffer_MB:10.1 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_72 | allocs:35264 alloc_MB:2.4 buffer_MB:2.9 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_80 | allocs:14920 alloc_MB:1.1 buffer_MB:1.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_88 | allocs:5582 alloc_MB:0.5 buffer_MB:0.6 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_104 | allocs:8075 alloc_MB:0.8 buffer_MB:1.0 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_128 | allocs:8892 alloc_MB:1.1 buffer_MB:1.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_160 | allocs:17614 alloc_MB:2.7 buffer_MB:3.0 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_200 | allocs:30454 alloc_MB:5.8 buffer_MB:6.9 cached_buffer_MB:0.6 |
| Alloc_variable_bucket_248 | allocs:4875 alloc_MB:1.2 buffer_MB:1.5 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_312 | allocs:371 alloc_MB:0.1 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_384 | allocs:30 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_480 | allocs:11 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_600 | allocs:57 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_752 | allocs:62 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_936 | allocs:42 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1168 | allocs:106 alloc_MB:0.1 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1480 | allocs:126 alloc_MB:0.2 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1832 | allocs:0 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_2288 | allocs:1 alloc_MB:0.0 buffer_MB:0.2 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_2832 | allocs:33 alloc_MB:0.1 buffer_MB:1.1 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_3528 | allocs:16 alloc_MB:0.1 buffer_MB:0.5 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_4504 | allocs:49 alloc_MB:0.2 buffer_MB:0.8 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_5680 | allocs:66 alloc_MB:0.4 buffer_MB:1.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_6224 | allocs:30 alloc_MB:0.2 buffer_MB:1.0 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_7264 | allocs:94 alloc_MB:0.7 buffer_MB:1.5 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_9344 | allocs:70 alloc_MB:0.6 buffer_MB:2.6 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_11896 | allocs:14 alloc_MB:0.2 buffer_MB:2.4 cached_buffer_MB:1.2 |
| Alloc_variable_bucket_14544 | allocs:7 alloc_MB:0.1 buffer_MB:2.4 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_18696 | allocs:18 alloc_MB:0.3 buffer_MB:3.2 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_21816 | allocs:4 alloc_MB:0.1 buffer_MB:0.4 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_26184 | allocs:6 alloc_MB:0.1 buffer_MB:0.9 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_32728 | allocs:13 alloc_MB:0.4 buffer_MB:2.4 cached_buffer_MB:1.4 |
| Alloc_variable_bucket_43648 | allocs:12 alloc_MB:0.5 buffer_MB:1.4 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_65472 | allocs:7 alloc_MB:0.4 buffer_MB:2.8 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_130960 | allocs:3 alloc_MB:0.4 buffer_MB:2.2 cached_buffer_MB:1.9 |
| Alloc_variable_cached_buffers | 21.4 MB |
| Alloc_variable_allocated | 438.7 MB |
| Successful_read_queries | 9048 |
| Successful_write_queries | 19096 |
| Failed_read_queries | 0 |
| Failed_write_queries | 4 |
| Rows_returned_by_reads | 75939 |
| Rows_affected_by_writes | 245 |
| Execution_time_of_reads | 7864 ms |
| Execution_time_of_write | 180311 ms |
| Transaction_buffer_wait_time | 0 ms |
| Transaction_log_flush_wait_time | 0 ms |
| Row_lock_wait_time | 0 ms |
| Ssl_accept_renegotiates | 0 |
| Ssl_accepts | 0 |
| Ssl_callback_cache_hits | 0 |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_ctx_verify_depth | 18446744073709551615 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_default_timeout | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_finished_connects | 0 |
| Ssl_session_cache_hits | 0 |
| Ssl_session_cache_misses | 0 |
| Ssl_session_cache_overflows | 0 |
| Ssl_session_cache_size | 20480 |
| Ssl_session_cache_timeouts | 0 |
| Ssl_sessions_reused | 0 |
| Ssl_used_session_cache_entries | 0 |
| Ssl_verify_depth | 0 |
| Ssl_verify_mode | 0 |
| Ssl_cipher | |
| Ssl_cipher_list | |
| Ssl_version | |
| Ssl_session_cache_mode | SERVER |
+-------------------------------------+------------------------------------------------------------------------+
From the status output, we can see:
10GB total memory on the leaf node
7GB Alloc_durability_large
You can see what these variables mean here: https://help.memsql.com/hc/en-us/articles/115001091386-What-Is-Using-Memory-on-My-Leaves-
Most interesting is the large amount in Alloc_durability_large, which is unusual. Do you have a large number of databases and/or partitions? (You can check by counting the number of rows in SHOW DATABASES EXTENDED on the leaf nodoe.) Each will require a fixed amount of transaction buffer memory (default is 64 MB).
I have a following statement and it generates the mentioned output by averaging data within every 20 minutes of range.
Statement :
SELECT record_no, date_time,
ROUND(AVG(UNIX_TIMESTAMP(date_time))) AS time_value,
ROUND(AVG(ph1_active_power),4) AS p1,
ROUND(AVG(ph2_active_power),4) AS p2,
ROUND(AVG(ph3_active_power),4) AS p3
FROM powerpro1
GROUP BY date_time DIV 2000
Portion of the output
+-----------+---------------------+------------+---------+----------+----------+
| record_no | date_time | time_value | p1 | p2 | p3 |
+-----------+---------------------+------------+---------+----------+----------+
| 1 | 2014-12-01 00:00:00 | 1417372770 | 72.6242 | -68.7428 | -72.6242 |
| 21 | 2014-12-01 00:20:00 | 1417373970 | 71.6624 | -69.7448 | -71.6624 |
| 41 | 2014-12-01 00:40:00 | 1417375170 | 70.6869 | -70.7333 | -70.6869 |
| 61 | 2014-12-01 01:00:00 | 1417376370 | 69.6977 | -71.7082 | -69.6977 |
| 81 | 2014-12-01 01:20:00 | 1417377570 | 68.6952 | -72.6692 | -68.6952 |
| 101 | 2014-12-01 01:40:00 | 1417378770 | 67.6794 | -73.6162 | -67.6794 |
| 121 | 2014-12-01 02:00:00 | 1417379970 | 66.6505 | -74.549 | -66.6505 |
| 141 | 2014-12-01 02:20:00 | 1417381200 | 65.5825 | -75.4901 | -65.5825 |
+-----------+---------------------+------------+---------+----------+----------+
According to the no of records in the table named "powerpro1", the above query selects 1368 records when the executing. (May be increased in the future when receiving new records)
My requirement is to create a highchart using time_value for the x-axis and p1, p1 and p3 for the y-axis. But I needs to limit the no of points in the x-axis.
Can anyone like to help me to show this 1368 points by 1000 points in the chart
Unfortunately have no this kind of apporixmation, only in reverse order (I mean datagrouping, if you have 100 points, return i.e 10). So you need to calcualte it on your own and push to your data all 1000 points.
I use the following code to copy dbf/mdx files from one folder to another:
procedure TfrmMain.MyCopyFile(S1, S2: string);
begin
if not FileExists(S2) then
CopyFile(PCHAR(S1), PCHAR(S2), true)
else
if Application.MessageBox(PCHAR('Overwrite existing file ' + S2 + '?'), 'File exists in folder',MB_YESNO + MB_DEFBUTTON1) = IDYES
then CopyFile(PCHAR(S1), PCHAR(S2), false)
end;
The code works fine when table name stays the same.
If I change the name of the table:
MyCopyFile(CurPath + '\orders.dbf', NewPath + '\ordly.dbf');
MyCopyFile(CurPath + '\orders.mdx', NewPath + '\ordly.mdx');
When I try to open ordly.dbf I get an error message:
Corrupt table/index header.
File: C:\DATA\2011\ORDLY.MDX
the problem is due to the mdx format stores inside the name of the data file associated (table name). because that when you rename a mdxfile the index still points to the old name of data file.
check this link to see the structure of the mdx file.
The Structure of Multiple Index files (*.mdx)
0 | Version number *1| ^
|-----------------------| |
1 | Date of creation | |
2 | YYMMDD | |
3 | | |
|-----------------------| |
4 | Data file name | File
5 | (no extension) | Header
: : |
: : |
19 | | |
|-----------------------| |
20 | Block size | |
| | |
|-----------------------| |
22 | Block size adder N | |
| | |
|-----------------------| |
24 | Production index flag | |
|-----------------------| |
25 | No. of entries in tag | | *2
|-----------------------| |
26 | Length of tag | | *3
|-----------------------| |
27 | (Reserved) | |
|-----------------------| |
28 | No.of tags in use | |
| | |
|-----------------------| |
30 | (Reserved) | |
| | |
|-----------------------| |
32 | No.of pages in tagfile| |
| | |
| | |
35 | | |
|-----------------------| |
36 | Pointer to first free | |
| page | |
| | |
39 | | |
|-----------------------| |
40 | No.of block available | |
| | |
| | |
43 | | |
|-----------------------| |
44 | Date of last update | |
| YYMMDD | |
46 | | |
|-----------------------| |
47 | (Reserved) | |
|-----------------------| |
48 | (Garbage) | |
: : |
: : |
| | | ___|=======================|
543| | _V___ / 0 | Tag header page no. |
|-----------------------| | / | |
544| Tag table entries | Tag / | |
| | Table | 3 | |
:.......................: | | |-----------------------| Tag
: : | | 4 | Tag name | table
:.......................: | | : :
: : | / : :
: : | / | |
:.......................:__|_/ 14 | |
: : | |-----------------------|
: : | 15 | Key format *4 |
: : | |-----------------------|
:.......................:__|_ 16 | Forward tag thread (<)|
: : | \ |-----------------------|
: : | \ 17 | Forward tag thread (>)|
: : | \ |-----------------------|
: : | | 18 | Backward tag thread *5|
| | | | |-----------------------|
| | | | 19 | (Reserved) |
M*N| |__V__ | |-----------------------|
|=======================| ^ | 20 | Key type *6 |
0| Pointer to root page | | | |-----------------------|
| | | | 21 | (Reserved) |
| | | | : :
3| | | | : :
|-----------------------| | | 31 | |
4| File size in pages | Tag | |-----------------------|
| | header| 32 | (Garbage) |
| | | | : :
7| | | | | |
|-----------------------| | \ N | |
8| Key format *7 | | \____|=======================|
|-----------------------| |
9| Key type *8 | |
|-----------------------| |
10| (Reserved) | |
| | |
|-----------------------| |
12| Index key length *9 | |
| | |
|-----------------------| |
14| Max.no.of keys/page | |
| | |
|-----------------------| |
16| Secondary key type *10| |
| | |
|-----------------------| |
18| Index key item length | |
| | |
|-----------------------| |
20| (Reserved) | |
| | |
| | |
|-----------------------| |
23| Unique flag | |
|-----------------------| |
| | |
: : |
: :__V__
N*M|=======================|