What I Learned Over My First Production Weekend

Recently, I got to actively participate in production. Before, I would shadow or watch the more experienced employees deliver while I sat back. Other times, I wouldn’t sit over a production weekend at all.

This time, I got to call a GET Mule service coded by an absent Mule developer, which means I was given an HTTP URL, headers, and a list of surrogate keys since there was a URI parameter. I also had to generate my own production token.

What my non-technical experience was like:

  1. I was really sleepy since I had a full Saturday, with my validation shift was on Sunday 1am. I ended up staying glued to my laptop until 6am, and then I had to wake up at 7:30am to explain my findings to the business.
  2. I got more help from senior developers than I had expected. I was stressed because I wasn’t sure if my production surrogate keys would actually be valid in production when I called the service I was assigned. I was pleased to get help on logistics, but I had prepared a bunch which helped me on the big day.
  3. Jumping on calls with the senior developers made me feel included and impactful, especially when I had the task of calling the services repeatedly with different surrogate keys and announcing the error codes or whether or not I spotted a successful result.

I also learned some technical lessons about validating services in production.

  1. Keep trying to call the service over time even if it doesn’t work at first. Just because validation doesn’t work at first doesn’t mean it won’t work later. When I was calling a GET request with an HTTP URL, I wasn’t getting a response at all. However, later, the developers fixed the environment and then I was getting a response. I learned from the senior developers that I have to keep trying and calling the middleware service over time, even though it doesn’t work at first. In the worst case, where you get no response for a service you are calling, you will be able to give an accurate timestamp for all the times you tried something, which is better than saying “it doesn’t work”.
  2. If the service call works and then doesn’t, don’t get excited if it works again because it might not work again.

I’ve coded pieces that made it to production, and I’ve validated production code. Hopefully, one day I will get to validate my own code in production.

Leave a Reply

Your email address will not be published. Required fields are marked *