I have registered my ESP32 as a thing on AWS IoT and downloaded its respective certificate and public & private keys. Also verified that those connect properly via the following command in my terminal:
openssl s_client -connect host.iot.region.amazonaws.com:8443 -CAfile AmazonRootCA1.pem -cert certificate.pem.crt -key private.pem.key
This is my (main.py) simple code to connect to AWS IoT using MicroPython
import machine
from network import WLAN
import network
from umqtt.simple import MQTTClient
# AWS endpoint parameters.
HOST = b'HOST' # ex: b'abcdefg1234567'
REGION = b'REGION' # ex: b'us-east-1'
CLIENT_ID = "CLIENT_ID" # Should be unique for each device connected.
AWS_ENDPOINT = b'%s.iot.%s.amazonaws.com' % (HOST, REGION)
keyfile = '/certs/private.pem.key'
with open(keyfile, 'r') as f:
key = f.read()
certfile = "/certs/certificate.pem.crt"
with open(certfile, 'r') as f:
cert = f.read()
# SSL certificates.
SSL_PARAMS = {'key': key,'cert': cert, 'server_side': False}
# Setup WiFi connection.
wlan = network.WLAN( network.STA_IF )
wlan.active( True )
wlan.connect( "SSID", "PASSWORD" )
while not wlan.isconnected():
machine.idle()
# Connect to MQTT broker.
mqtt = MQTTClient( CLIENT_ID, AWS_ENDPOINT, port = 8883, keepalive = 10000, ssl = True, ssl_params = SSL_PARAMS )
mqtt.connect()
# Publish a test MQTT message.
mqtt.publish( topic = 'test', msg = 'hello world', qos = 0 )
But I get this error when I try to connect:
(-17168, 'MBEDTLS_ERR_RSA_PRIVATE_FAILED+MBEDTLS_ERR_MPI_ALLOC_FAILED')
After much effort I got this to work. I had to use an idf3 MicroPython binary,
esp32-idf3-20191220-v1.12.bin
idf4 binaries and idf3 later than v1.12 don't work. There is a problem with not enough heap and memory allocation problems.
----------- EDIT -----------
News update! The new v1.15 release of MicroPython based on idf4 works with AWS MQTT for IoT.
Related
I am running a local Mosquitto (MQTT) Broker that connects to a remote Mosquitto Broker using the build in MQTT Bridge functionality of Mosquitto. My mosquitto.conf looks like this:
# =================================================================
# Listeners
# =================================================================
listener 1883
# =================================================================
# Security
# =================================================================
allow_anonymous true
# =================================================================
# Bridges
# =================================================================
connection myConnectionName
address <<Remote Broker IP>>:1883
remote_username <<Remote Broker Username>>
remote_password <<Remote Broker Password>>
topic mytopic/# out 1 "" B2/
bridge_protocol_version mqttv50
cleansession false
bridge_attempt_unsubscribe true
upgrade_outgoing_qos true
max_queued_messages 5000
For testing I run a MqttPublisher using a C# console application which uses the MQTTnet library (version 3) and a MqttSubsbriber (also C# console application with MqttNet).
Now I want the Publisher to publish MQTT messages with User Properties (introduced by MQTT 5).
I build the message like this:
using MQTTnet;
using MQTTnet.Client;
using MQTTnet.Client.Options;
class Program
{
static void Main()
{
// Create a new MQTT client instance
var factory = new MqttFactory();
var mqttClient = factory.CreateMqttClient();
// Setup the options for the MQTT client
var options = new MqttClientOptionsBuilder()
.WithClientId("MqttPublisher")
.WithTcpServer("localhost", 1883)
.WithProtocolVersion(MQTTnet.Formatter.MqttProtocolVersion.V500)
.WithCleanSession()
.Build();
mqttClient.ConnectAsync(options).Wait();
var i = 0;
while (true)
{
Console.WriteLine("Client connected: " + mqttClient.IsConnected);
var message = new MqttApplicationMessageBuilder()
.WithTopic("mytopic/test")
.WithUserProperty("UPTest","Hi UP")
.WithPayload("Hello World: " + i)
.Build();
mqttClient.PublishAsync(message).Wait();
Console.WriteLine("Published message with payload: " + System.Text.Encoding.UTF8.GetString(message.Payload));
i++;
System.Threading.Thread.Sleep(1000);
}
mqttClient.DisconnectAsync().Wait();
}
}
With the subscriber (also with WithProtocolVersion(MQTTnet.Formatter.MqttProtocolVersion.V500) if I subscribe to the topic I get all the messages and I can read the MQTTnet.MqttApplicationMessage like shown in the following screenshot:
The messages are also published to the remote MQTT Broker due to the MQTT Bride configured. However, if I subscribe to the remote Broker with my MqttSubscriber, I am not getting the User Properties anymore:
Is there any way to configure the Mosquitto Bridge that also the user properties are send? I cant find a way and any help and comments are appreciated.
Thanks
Joshua
Using mosqutto 2.0.15 I have verified that MQTTv5 message properties are passed over the bridge.
Broker one.conf
listener 1883
allow_anonymous true
Broker two.conf
listener 1884
allow_anonymous true
connection one
address 127.0.0.1:1883
topic foo/# both 0
bridge_protocol_version mqttv50
Subscriber connected to Broker two
$ mosquitto_sub -t 'foo/#' -V mqttv5 -p 1884 -F "%t %P %p"
foo/bar foo:bar ben
Publisher connected to Broker one
$ mosquitto_pub -D PUBLISH user-property foo bar -t 'foo/bar' -m ben -p 1883
As you can see the the %P in the output format for the subscriber is outputting the user property foo with a value of bar when subscribed over the bridge.
A few days ago I bought a Raspberry Pi Pico W and I am trying to connect it to AWS Iot Core. However, I get stuck when connecting to AWS.
I have gone throught the following steps:
installed Micropython on the Raspberry Pi Pico W, this works!
Created an AWS account.
registered a "Thing" at IoT Core
Downloaded the corresponding certificates & keys: (Certificates: AmazonRootCA1.pem, certificate.pem.crt, Keys: public.pem.key & private.pem.key)
created a policy, which allows the policy all actions. At AWS this policy is activated and has the form:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
Attached the certificate to the policy.
Attached the registered Thing to the policy.
transfered the keys and the certificates to the Raspberry Pi Pico W.
Hereafter, I use Thonny to write the following code trying to connect the raspberry pi with AWS.
import time
import network
import urequests
SSID = "XXXX"
PASSWORD = "XXXX"
print("start connecting")
wlan = network.WLAN(network.STA_IF)
wlan.active(True)
wlan.connect(SSID, PASSWORD)
print("Connected:", wlan.isconnected())
import sys
import machine
import argparse
from umqtt.simple import MQTTClient
clientId = 'client1'
AWS_ENDPOINT = 'XXXXX.amazonaws.com'
PORT = 8883
certfile = '/certificate.pem.crt'
with open(certfile, 'r') as f:
cert = f.read()
keyfile = '/private.pem.key'
with open(keyfile, 'r') as f:
key = f.read()
print("Key and Certificate files Loaded")
SSL_PARAMS = {'key': key, 'cert': cert, 'server_side': False}
client = MQTTClient(clientId, AWS_ENDPOINT, port=PORT, keepalive=10000, ssl=True, ssl_params=SSL_PARAMS)
print("Client created")
client.connect()
The program runs until:
client.connect()
And gives me the following error:
Traceback (most recent call last):
File "<stdin>", line 38, in <module>
File "/lib/umqtt/simple.py", line 61, in connect
ValueError: invalid key
(line 38 being the client.connect() statement)
The format of the private.pem.key is:
-----BEGIN RSA PRIVATE KEY-----
[block of characters]
-----END RSA PRIVATE KEY-----
I there anyone who knows what I am doing wrong? Thank you in advance for the help!! :)
Kind regards, Benjamin
Sorry - late to the party.
If you haven't got this working you should try to convert the key & cert files to .DER format with OpenSSL
openssl x509 -in certificate.pem.crt -out certificate.der -outform DER
openssl rsa -in private.pem.key -out private.key.der -outform DER
`certfile = '/certificate.der'`
`with open(certfile, 'r') as f:`
`cert = f.read()`
`keyfile = '/private.pem.der'`
`with open(keyfile, 'r') as f:`
`key = f.read()`
I've looked through the list of possible solutions, but I don't see this problem, here it is.
I had been using smtp for years for my crontab entry to provide status updates via email. Then it quit this week, and I was unable to fix it. Then I saw that it had become orphaned, and the suggestion was to move to msmtp. So I downloaded and installed it on my Ubuntu 18.10 system.
I'm trying to send email to my myaccount#gmail.com account.
It appears that I'm communicating properly with the gmail smtp server, as the debug below show. But it always gets a TLS Timeout.
I also don't understand why I have multiple EHLO entries. My system does not have a DNS domain name, so that I'm not sure what to put here; localhost seems to be working OK. Also, my Thunderbird emailer is working correctly with gmail.
Here's the debug output:
echo "Hello there" | msmtp --debug myaccount#gmail.com >/tmp/msmtpOut.txt
ignoring system configuration file /etc/msmtprc: No such file or directory
loaded user configuration file /home/myhome/.msmtprc
falling back to default account
using account default from /home/myhome/.msmtprc
host = smtp.gmail.com
port = 587
proxy host = (not set)
proxy port = 0
timeout = off
protocol = smtp
domain = localhost
auth = choose
user = myaccount
password = *
passwordeval = (not set)
ntlmdomain = (not set)
tls = on
tls_starttls = on
tls_trust_file = /etc/ssl/certs/ca-certificates.crt
tls_crl_file = (not set)
tls_fingerprint = (not set)
tls_key_file = (not set)
tls_cert_file = (not set)
tls_certcheck = on
tls_min_dh_prime_bits = (not set)
tls_priorities = (not set)
auto_from = off
maildomain = (not set)
from = myaccount#gmail.com
add_missing_from_header = on
dsn_notify = (not set)
dsn_return = (not set)
logfile = (not set)
syslog = (not set)
aliases = (not set)
reading recipients from the command line
<-- 220 smtp.gmail.com ESMTP 4sm116524ywc.22 - gsmtp
--> EHLO localhost
<-- 250-smtp.gmail.com at your service, [71.56.87.81]
<-- 250-SIZE 35882577
<-- 250-8BITMIME
<-- 250-STARTTLS
<-- 250-ENHANCEDSTATUSCODES
<-- 250-PIPELINING
<-- 250-CHUNKING
<-- 250 SMTPUTF8
--> STARTTLS
<-- 220 2.0.0 Ready to start TLS
TLS certificate information:
Owner:
Common Name: smtp.gmail.com
Organization: Google LLC
Locality: Mountain View
State or Province: California
Country: US
Issuer:
Common Name: Google Internet Authority G3
Organization: Google Trust Services
Country: US
Validity:
Activation time: Tue 21 May 2019 04:48:45 PM EDT
Expiration time: Tue 13 Aug 2019 04:32:00 PM EDT
Fingerprints:
SHA256: C7:78:B6:D6:4E:3E:2B:2F:08:6D:A4:84:E6:1D:87:8E:A1:DF:54:D2:AB:79:AC:A6:BB:50:E5:5D:EC:B4:20:4C
SHA1 (deprecated): 39:C5:E5:40:64:37:17:25:17:7F:E8:BA:20:F4:70:F4:FE:22:70:22
--> EHLO localhost
msmtp: cannot read from TLS connection: the operation timed out
msmtp: could not send mail (account default from /home/myhome/.msmtprc)
Build msmtp using --with-tls=openssl to solve the problem.
As regards as the EHLO command sent twice the RFC3207 states:
The server MUST discard any knowledge
obtained from the client, such as the argument to the EHLO command,
which was not obtained from the TLS negotiation itself. The client
MUST discard any knowledge obtained from the server, such as the list
of SMTP service extensions, which was not obtained from the TLS
negotiation itself. The client SHOULD send an EHLO command as the
first command after a successful TLS negotiation.
So that is the normal behaviour.
I am trying to setup AWS IoT in Pi on port 443 using Paho MQTT .
As AWS document (https://docs.aws.amazon.com/iot/latest/developerguide/protocols.html) mentioned that
Clients wishing to connect using MQTT with X.509 Client Certificate authentication on port 443 must implement the Application Layer Protocol Negotiation (ALPN) TLS extension and pass x-amzn-mqtt-ca as the ProtocolName in the ProtocolNameList.
I actually don't know how to achieve it properly in Paho MQTT (https://github.com/eclipse/paho.mqtt.python)
What I tried to do (mqtt_apln.py)
import sys
import ssl
import time
import datetime
import logging, traceback
import paho.mqtt.client as mqtt
MQTT_TOPIC = "topictest"
MQTT_MSG = "hello MQTT"
IoT_protocol_name = "x-amzn-mqtt-ca"
aws_iot_endpoint = "xxxxxxx.iot.eu-west-1.amazonaws.com"
url = "https://{}".format(aws_iot_endpoint)
ca = ".xxxxx/rootCA.pem"
cert = ".xxxxx/xxxxx-certificate.pem.crt"
private = ".xxxxx/xxxxxx-private.pem.key"
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler(sys.stdout)
log_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(log_format)
logger.addHandler(handler)
# Define on connect event function
# We shall subscribe to our Topic in this function
def on_connect(mosq, obj, rc):
mqttc.subscribe(MQTT_TOPIC, 0)
# Define on_message event function.
# This function will be invoked every time,
# a new message arrives for the subscribed topic
def on_message(mosq, obj, msg):
print "Topic: " + str(msg.topic)
print "QoS: " + str(msg.qos)
print "Payload: " + str(msg.payload)
def on_subscribe(mosq, obj, mid, granted_qos):
print("Subscribed to Topic: " +
MQTT_MSG + " with QoS: " + str(granted_qos))
def ssl_alpn():
try:
#debug print opnessl version
logger.info("open ssl version:{}".format(ssl.OPENSSL_VERSION))
ssl_context = ssl.create_default_context()
ssl_context.set_alpn_protocols([IoT_protocol_name])
ssl_context.load_verify_locations(cafile=ca)
ssl_context.load_cert_chain(certfile=cert, keyfile=private)
return ssl_context
except Exception as e:
print("exception ssl_alpn()")
raise e
mqttc = mqtt.Client()
# Assign event callbacks
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
ssl_context= ssl_alpn()
mqttc.tls_set_context(context=ssl_context)
logger.info("start connect")
mqttc.connect(aws_iot_endpoint, port=443)
logger.info("connect success")
mqttc.loop_start()
In Pi, I installed python 2.7.14 and paho-mqtt
But When I run python mqtt_apln.py, it shows error: ImportError: No module named paho.mqtt.client
Any suggestion is appreciated
I think there are two things going on here. First, a pip install paho-mqtt should make the package active for the current referenced python. For instance, under a 3.6.2 virtualenv should return:
$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your
pip.conf under the [list] section) to disable this warning.
paho-mqtt (1.3.1)
pip (9.0.1)
setuptools (28.8.0)
How did you install the paho-mqtt package, via an apt package or directly with pip? Personally I virtualenv everything or include the package within the application directory via pip install package_name -t . to reference the current working directory.
From there it's working with the ALPN configuration. I reduced your code to just publish to the test topic on my end-point and used the AWS IoT console-->Test to subscribe to the test topic. For both python 2.7.12 and 3.6.2 I successfully received messages.
The main changes were to remove the callbacks, place a mqttc.publish followed by a time.sleep(3) to give the thread time to publish, then closed the connection.
Here is the code paired down for the publish only:
import sys
import ssl
import time
import datetime
import logging, traceback
import paho.mqtt.client as mqtt
MQTT_TOPIC = "topictest"
MQTT_MSG = "hello MQTT"
IoT_protocol_name = "x-amzn-mqtt-ca"
aws_iot_endpoint = "xxxxxx.iot.us-east-1.amazonaws.com"
ca = "ca.pem"
cert = "xxxx-certificate.pem.crt"
private = "xxxx-private.pem.key"
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler(sys.stdout)
log_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(log_format)
logger.addHandler(handler)
def ssl_alpn():
try:
#debug print opnessl version
logger.info("open ssl version:{}".format(ssl.OPENSSL_VERSION))
ssl_context = ssl.create_default_context()
ssl_context.set_alpn_protocols([IoT_protocol_name])
ssl_context.load_verify_locations(cafile=ca)
ssl_context.load_cert_chain(certfile=cert, keyfile=private)
return ssl_context
except Exception as e:
print("exception ssl_alpn()")
raise e
mqttc = mqtt.Client()
ssl_context= ssl_alpn()
mqttc.tls_set_context(context=ssl_context)
logger.info("start connect")
mqttc.connect(aws_iot_endpoint, port=443)
logger.info("connect success")
mqttc.loop_start()
# After loop start publish and wait for message to be sent.
# Hard coded delay but would normally tie into event loop
# or on_publish() CB
# JSON payload because, pretty
mqttc.publish('test', '{"foo": "bar"}')
time.sleep(3)
mqttc.loop_stop()
Please let me know if that works for you? It's awesome that AWS now supports MQTT connections on port 443 without having to utilize websockets (and the need for SigV4 credentials).
I ran into the same issue regarding the use of paho-mqtt with aws IoT core.
https://aws.amazon.com/de/blogs/iot/how-to-implement-mqtt-with-tls-client-authentication-on-port-443-from-client-devices-python/
The tutorial uses no clientID. Depending on your security rules you have to deliver a client ID to be able to connect properly. Here an SDK-example ruleset where only the clients "sdk-java", "basicPubSub" and "sdk-nodejs-*" are allowed to connect.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Publish",
"iot:Receive"
],
"Resource": [
"arn:aws:iot:eu-central-1:036954049003:topic/sdk/test/java",
"arn:aws:iot:eu-central-1:036954049003:topic/sdk/test/Python",
"arn:aws:iot:eu-central-1:036954049003:topic/topic_1",
"arn:aws:iot:eu-central-1:036954049003:topic/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Subscribe"
],
"Resource": [
"arn:aws:iot:eu-central-1:036954049003:topicfilter/sdk/test/java",
"arn:aws:iot:eu-central-1:036954049003:topicfilter/sdk/test/Python",
"arn:aws:iot:eu-central-1:036954049003:topicfilter/topic_1",
"arn:aws:iot:eu-central-1:036954049003:topicfilter/topic_2"
]
},
{
"Effect": "Allow",
"Action": [
"iot:Connect"
],
"Resource": [
"arn:aws:iot:eu-central-1:036954049003:client/sdk-java",
"arn:aws:iot:eu-central-1:036954049003:client/basicPubSub",
"arn:aws:iot:eu-central-1:036954049003:client/sdk-nodejs-*"
]
}
]
}
To allow connection in case you have permits based on clientid change this line:
mqttc = mqtt.Client(client_id=MYCLIENTID)
MYCLIENTID is one of the three "sdk-java", "basicPubSub" or "sdk-nodejs-*" for this example.
I have a Rpi-A connected to internet via 3G surf-stick and Rpi-B connected to internet via a WiFi hotspot. Rpi-A has a public ip address and also ports 1883 and 8883 are open. Both raspberry Pi's are on different networks. I am attempting to send binary data using MQTT from Rpi-B to Rpi-A.
update: I used the below code to test the MQTT connection. replacing XX.XX.XX.XX with public IP of raspberry Pi. Still I end up getting this error--->
error: [Errno 10060] A connection attempt failed because the connected
party did not properly respond after a period of time, or established
connection failed because connected host has failed to respond.
what might be the possible reason for this error ? Is there anything missing in my code
import paho.mqtt.client as mqtt
import time
def on_connect(client, userdata, flags, rc):
print("Connected with result code " + str(rc))
client = mqtt.Client()
client.on_connect = on_connect
client.connect("xx.xx.xxx.x", 1883, 60)
client.loop_start()
while True:
time.sleep(2)
client.publish('Due_0.72/cmd/in','hello')
print "publish.."