cody

Author

This was an interesting challenge I ran into recently and has been the first time in a while that I’ve felt that itch to solve a problem just because I wanted to see if I could do it.

It’s been (checks notes) about 7 years since the last time I wrote anything related to ruby and probably 5 years since I’ve given any talks or presentations.

I don’t think I realized how burnt out I was after a few tumultuous years both personally and professionally. It was great to feel like I had to solve a problem just for fun and was even compelled to write about it.

That being said, ya boy is rusty, so bear with me.

Recently, I’ve been doing some work on a side project that uses Twilio for sending and receiving text messages. I’ve always thought of SMS as a really powerful tool to add to any app’s arsenal since it leverages something users already use multiple times per day. In a recent report, Twilio even reported that 96% of messages are read within three minutes of receipt, and 90% within three seconds!

This post won’t go into detail on sending text messages. If you’d like some more information on how to send text messages, the twilio-ruby gem has some examples.

Twilio phone numbers have webhooks for both Voice and SMS. These webhooks are called whenever your Twilio phone number gets an inbound text or phone call.

Incoming message example

Both channels have a fallback URL if the first request to the webhook fails.

Sending Webhook Requests To Your Local Environment

If you’ve ever developed features from a third-party webhook or service with your local environment, you probably did it by exposing your local environment to the internet with something like ngrok.

If you haven’t heard of ngrok, it’s one of my favorite tools that I’ve been integrating into more workflow more and more. You can read more about ngrok here.

Running Ngrok In Your Procfile

With Rails 7 embracing using foreman and Procfiles for starting and running services, I’ve added ngrok as its own process so it’s always available.

example Procfile.dev

# Procfile.dev
web: bin/rails server -p $PORT
css: yarn build:css --watch
js: yarn build --reload
worker: bundle exec sidekiq
stripe: stripe listen --forward-to $PORT/webhooks/stripe
ngrok: ngrok http --log=stdout $PORT

Updating Twilio Webhook URLs

With ngrok running, there’s now a URL to point your webhooks to. One option for updating your webhook URLs is grabbing your ngrok URL, going into the Twilio console and update the webhook URLs for voice and sms.

That was fine for me…for a while.

My main issue with this setup is whenever I restart my services, the ngrok URL changes and needs to be updated again. That can make for a lot of updates when starting and stopping your services and errors when forgetting to update the URLs each time.

Also, if that’s something you’re not doing very often, it’s easy to forget how to find the section to update the webhook URLs.

That’s when I felt the urge to find a way to automate that whenever I needed it.

After some digging in the Twilio documentation, I found some options to set the webhooks URLs through their ruby gem

A little more digging helped me find an endpoint for ngrok you can query to get it’s public url.

Now we’re cooking with gas. With the URL of our local ngrok instance and a way to programmatically update the webhook URLs, we have the 2 pieces we need.

The next step is to decide how to execute that code when we need updates. For me, that was a rake task. That gives me a nice repeatable way to update the URLs whenever I need to.

Rake Task For Updating URLs

Before I start going into more detail, here’s the code for the rake task. The dependencies for the rake task are a running ngrok instance, the twilio-ruby gem (I’m using v5.67.1), and if you’d like a pop of color in your terminal to make things easier to spot, the colorize gem

Here is an example of the rake task

namespace :development do
  desc "updates twilio webhook urls to ngrok url"
  task set_twilio_webhooks: :environment do
    return unless Rails.env.development?
    begin
      response = JSON.parse(Net::HTTP.get(URI("http://localhost:4040/api/tunnels")))
      ngrok_url = response.dig("tunnels", 1, "public_url")
    rescue Errno::ECONNREFUSED
      puts "*******NGROK not running*******".yellow
    end
    if ngrok_url
      puts ngrok_url.green
      account_sid = Rails.application.credentials.dig(:twilio)[:account_sid]
      auth_token = Rails.application.credentials.dig(:twilio)[:auth_token]
      development_incoming_phone_number = Rails.application.credentials.dig(:twilio)[:auth_token]
      twilio_client = Twilio::Client.new(account_sid, auth_token)
      twilio_client.api.v2010.accounts(account_sid).incoming_phone_numbers(developemnt_incoming_phone_number)
        .update(sms_url: "#{ngrok_url}/webhooks/twilio",
          sms_fallback_url: "#{ngrok_url}/webhooks/twilio_fallback",
          voice_url: "#{ngrok_url}/webhooks/twilio_voice",
          voice_fallback_url: "#{ngrok_url}/webhooks/twilio_voice_fallback")
      puts "Webhook URLs updated!".green
    rescue Twilio::REST::RestError => e
      puts e
    end
    end
  end
end

The Deets

Now, let’s dig a little deeper into what’s going on.

Inside the rake task, the first thing I’m doing is making sure we’re in the development environment and returning from the task if we’re not. Probably not required, but I thought it was a nice check to keep from accidentally updating information outside of our development environment.

After ensuring we’re in our dev environment, we need to do is find the URL of the running ngrok instance. Ngrok has an API endpoint running at port 4040 for the current tunnels. This endpoint returns an XML response with information about the current tunnels. There are 2 tunnels returned, and I grab the one with the name command_line (http) which is the second one returned.

Usually, my ruby HTTP library of choice is HTTParty but I wanted to set this up using the ruby Net::HTTP lib to keep from introducing another dependency.

After ensuring we’re in the development environment, Net::HTTP then makes a request to the ngrok endpoint and parses the XML response.

Here are some examples of the response it sends back.

Truncated Ngrok response

<tunnelListResource>
  <Tunnels>
  </Tunnels>
  <Tunnels>
  </Tunnels>
  <URI>/api/tunnels</URI>
</tunnelListResource>

Single Tunnel Object

<Tunnels>
  <Name>command_line</Name>
  <URI>/api/tunnels/command_line</URI>
  <PublicURL>https://7be0-72-42-102-194.ngrok.io</PublicURL>
  <Proto>https</Proto>
  <Config>
  <Addr>http://localhost:80</Addr>
  <Inspect>true</Inspect>
  </Config>
  <Metrics>
  <Conns>
  </Conns>
  <HTTP>
  <Count>0</Count>
  <Rate1>0</Rate1>
  <Rate5>0</Rate5>
  <Rate15>0</Rate15>
  <P50>0</P50>
  <P90>0</P90>
  <P95>0</P95>
  <P99>0</P99>
  </HTTP>
  </Metrics>
</Tunnels>

The rake task takes the parsed XML and grabs the 2nd Tunnel returned. Why you ask?

Honestly, it was most likely the first thing that worked from StackOverflow and I moved on.

The <PublicURL> for both tunnels seem to always be the same so it’s possible either would work.

If ngrok is not running, we’ll get an Errno::ECONNREFUSED error so we can rescue that error, and print some output in yellow to make it easy to know when something went wrong

Grab Ngrok Public Url

  begin
    response = JSON.parse(Net::HTTP.get(URI("http://localhost:4040/api/tunnels")))
    ngrok_url = response.dig("tunnels", 1, "public_url")
  rescue Errno::ECONNREFUSED
    puts "*******NGROK not running*******".yellow
  end

Now we have a repeatable and reliable way to grab our ngrok URL whenever it is running. We also get some colored output to give us a quick visual indicator that everything looks a-ok.

The next part is to programmatically update the Twilio webhook endpoints with our public URL for ngrok.

It took quite a bit of digging to find the endpoints needed in the twilio-ruby gem, but once I found them, it was smooth sailing from there.

Updating with the Twilio API

There are a few things we’ll need to update the webhook URLs for Twilio. The first thing we need is a client with our keys.

To create the Twilio client, you’ll need to get your account_sid and auth_token. You can find both of those by logging into Twilio. While you’re there, be sure to find the service id of your phone number. That’s required to target the correct phone number for the updates.

Twilio Updates

  account_sid = Rails.application.credentials.dig(:twilio)[:account_sid]
  auth_token = Rails.application.credentials.dig(:twilio)[:auth_token]
  development_incoming_phone_number = Rails.application.credentials.dig(:twilio)[:auth_token]
  twilio_client = Twilio::Client.new(account_sid, auth_token)
  twilio_client.api.v2010.accounts(account_sid).incoming_phone_numbers(developemnt_incoming_phone_number)
    .update(sms_url: "#{ngrok_url}/webhooks/twilio",
      sms_fallback_url: "#{ngrok_url}/webhooks/twilio_fallback",
      voice_url: "#{ngrok_url}/webhooks/twilio_voice",
      voice_fallback_url: "#{ngrok_url}/webhooks/twilio_voice_fallback")
  puts "Webhook URLs updated!".green
rescue Twilio::REST::RestError => e
  puts e
end

I included the Twilio client setup in the rake task, but that was mostly to have an example that would work as-is without additional configuration. I typically configure my Twilio client in an initializer to keep duplication to a minimum.

example config/initializers/twilio.rb file

Twilio.configure do |config|
  config.account_sid = Rails.application.credentials.dig(:twilio)[:account_sid]
  config.auth_token = Rails.application.credentials.dig(:twilio)[:auth_token]
end

With the client created, we can use the ngrok_url combined with the endpoint that points to your webhook handler and update our Twilio webhook URLs.

If everything goes as planned, we’ll output a success message in green.

If things don’t go as planned, we’re rescuing Twilio::REST::RestError and outputting the error.

Now, whenever we need to update our Twilio webhook URLs (or any other 3rd party service in the future) we can run:

$ bin/rails development:set_twilio_webhooks

Finding a way to automatically run ngrok in my Procfile and have a way to programmatically grab its public URL was a great find for my workflow. This is something that can be expanded to tons of other services you may like to work with in your development environment.

Let me know any other services you automate!