Can you trust your critical text message notifications to be delivered by your wireless carrier?

August 24, 2015 / 0 comments / in Support  / by Jacob Allred

Here is a scenario that I’ve seen crop up several times in the last few months:

A point enters the alarm state and an SMS notification has been sent, but several hours later no one has acknowledged it yet. Investigation begins into this issue and I notice that the notification service (GNS) has logged a successful send to my field techs’ wireless carrier’s email address, there are no errors reported, but no one has responded to the callout. I call the tech and he says he never received the notification.

What’s going on here? How could the service log a successful send, no errors are reported, yet no one receives a notification?

These are the same questions we’ve been asked in the last few months and my hope is to help all CygNet users understand what appears to be a change in the carrier’s administration of their email to SMS service.

How does the GNS send text messages to phones?

First, let’s start with how notifications from the GNS make their way to your server to a cellphone as a text message. It’s a simple process and is actually no different than sending an email notification. As a SCADA administrator, you simply set up the notification address to be sent to the desired phone number and set the domain to be the correct carrier domain (number@vtext.com or number@txt.att.net are two common examples). After properly configuring this, any notification will be sent as an email out to the configured domain. Once delivered, the wireless carrier will take the email notification and convert it to an SMS and finally deliver it to the correct recipient.

What are the reasons my notifications aren’t being delivered?

This is an easy question to answer, but a difficult one to understand. It has come to our attention that the wireless carriers have automated rules that can hinder or inhibit your ability to send notifications via the Email to SMS service. Once you’ve tripped a rule, the system will place you in a “Blacklisted” state.  Once you’re in a blacklisted state, you may not be able to send notifications of any kind (100% blacklisted) or possibly, your messages will be blocked for other reasons (content-based blacklist). From our experience with existing customers, your blacklisting may be comprised of the following:

• Notification address – Blocked based on the notification email (notification@company.com)
• Email domain – Blocked based on your @domain.com configured in the notification
• External IP Address – Blocked based on the external IP where the notifications originate
• Content – Blocked based on the content. One example was that the ack number listed in the SMS was “555-123-4567” changing the notification email to drop the hyphens worked around this blacklist rule “5551234567”.

How do the carriers flag me as blacklisted?

We’ve found some potential triggers for being blacklisted, but this is by no means an exhaustive list. On top of that, we’ve only worked with ATT and Verizon in determining the rules they use. Other carriers will have their own sets of rules. Based on the feedback we’ve gotten from ATT and Verizon, we’ve found the following are potential triggers for being blacklisted:

• Sending multiple notifications in rapid succession
• Send notifications with suspicious content

In every case, when asking the carriers for more details, we didn’t get much. The best we got regarding the rapid succession was “10 messages within 10 seconds” was a response on what “rapid succession” could be.

How do I avoid being blacklisted?

In speaking with the carriers, there are only two options available to you if you would like to try and avoid being blacklisted. These options are:

• Don’t get flagged by their system as a spammer
• Pay for their business messaging service

Both of these options have their advantages and disadvantages. Option 1 is business as usual, you don’t pay anything, but you may be blacklisted in the future if your system pushes a large number of notifications at one given time. Option 2 has a cost associated, but you’re guaranteed to not be blacklisted.

How do I troubleshoot this?

If you believe that you are experiencing this problem it is quite easy to verify that you are indeed blacklisted, however, there is no feedback loop from the wireless carrier that you are flagged.  Update one of your SMS based notifications to also include an email destination. Now simply send a test notification out. If your email is successful and the SMS is not, you have a problem with your wireless carrier. Keep in mind, since the wireless carriers can blacklist you entirely or by message contents, a simple “test” string in the body of your test SMS may be insufficient. To fully test, you should use a known failing message verbatim as the contents of your test SMS. If your email is successful and your SMS is not, your emails to the wireless carrier are potentially being dropped.

Conclusion

The carriers are making SMS notifications a risky business to be in. At any moment in time, you may hit an invisible ceiling which will cause your notifications to start failing silently. If you’re currently using SMS to communicate critical communications to field personnel, you may want to rethink that strategy or, call your carrier to voice your concerns and come up with a strategy moving forward.

I’d hate for your notifications to start silently failing on you as it has other customers.