Shaping a Full-Fledge Procedure by Streaming Salesforce Exception Emails

At Picnic, we use Salesforce to fulfil diverse purposes: from storing and maintaining data about the products we present to our customers, to handling any of their requests, questions, or feedback in a timely manner.

In that regard, Salesforce is similar to a multi-tool pocket knife. However, as the saying goes, “with great power comes great responsibility”, and having such a large range of functionalities always implies evaluating which ones should be used, improved, or ignored, depending on the various use cases a business might have.

Naturally, the more features we pick and use cases we have, the more challenging it can get to solve substantial problems. As a means to provide developers with a way to handle exceptions consistently, regardless of the requirements we may have or the features we may need in the future, we strived to figure out how we could proactively and efficiently track Salesforce issues to perpetually ensure the quality and accessibility of our services.

The default Salesforce way

In Salesforce, there are tons of different strategies to process and transform data (Triggers, Process Builders, Flows, Workflows, etc.) and the way Salesforce notifies us about unhandled errors is through the Apex Exception Email Notification feature: whenever an issue occurs, Salesforce sends an email containing a detailed stack trace of all processes that were executed before the exception arose.

When we first configured this feature, we decided that we would have Salesforce send emails to an email group, so that they would be automatically forwarded to our Salesforce administrators. The main advantages were that everyone would receive the exceptions in their own email address inboxes, and that we would avoid having an email account being shared across the company.

In order to prevent administrators from constantly checking their inboxes, we also added a hook between Gmail and Slack (the email service and the main communication platform we use at Picnic, respectively) to notify them via an alert in a channel as soon as an email was received.

Why it is inefficient

Despite the effectiveness of the Salesforce feature, we realised that the way we decided to use it ended up coming with massive disadvantages:

• All inboxes started to grow exponentially, which made finding other types of emails more tedious.
• As emails were sent to all administrators, it was challenging to determine whether someone else had already acknowledged an issue, unless it was communicated out loud.
• Only administrators were added to the contact group, excluding a majority of the Salesforce developers, who were only aware of explicitly reported issues.
• Solely based on emails, we could not easily create tickets in JIRA and fetch all the common stack traces related to a specific issue (Editor’s note: speaking from experience, the best way to waste time is to start browsing emails that have been received to gather clues from similar stack traces).
• The hook between Gmail and Slack that we initially added to simplify our jobs was in fact weakening our productivity.

Even if we love to build products and features in-house which are incisively shaped to fit our needs, we didn’t want to reinvent the wheel: in addition to the core being ready for use, implementing such a feature ourselves in Salesforce would have added too much complexity, altering the maintainability of our products. Therefore the main challenge was to discern how we could exploit the Apex Exception Email Notification functionality to significantly improve the Picnic Software developers’ course of action.

How we solved it

Earlier, we mentioned pain points brought by managing problems: lack of identifiable issues leading to follow-up deficiencies and tedious investigations. On top of that, we’re concerned about the environment so the fewer emails we have, the happier we are!

First of all, we created a dedicated account that would receive all emails containing the Salesforce exceptions in order to shrink the number of recipients to one and, as a consequence, we divided the total number of emails sent and received by 5.

You may recall that one of the reasons we forwarded emails to multiple inboxes was to avoid sharing a unique account. Although, now that we actually got rid of the superfluous inboxes and have a dedicated one, we clearly went against the initial flow. Rest assured: by choosing that path, we were aware that Security would naturally become a problem if we decided to keep the same workflow.

Previously, all administrators would receive an email from Salesforce but since we completely ruled out the option, we needed to ensure everyone can still examine the content of the emails while restricting their access to the account. To find the best way to achieve this, we had to dig into the different tools and systems directly (or indirectly) within our developers’ reach.

With the purpose of proactively reacting to issues, we currently use Sentry as one of the solutions to monitor our applications. Sentry is an error-monitoring tool Picnic developers use on a daily basis in order to easily identify and replicate bugs that occur in our systems, but also to create tickets on JIRA and link them to reported stack traces.

Based on what we use regularly, a viable approach would be to propagate the exceptions contained in the emails to Sentry, however, the Salesforce feature solely allows users to fill the email addresses to which the errors will be forwarded. Therefore, we needed to add a supplementary player which would be able to unify Salesforce, Gmail, and Sentry altogether.

Fortunately, we had the perfect binder for this purpose: the Salesforce Bridge. The Salesforce Bridge is a Java application dedicated to acting as a middleman between the Picnic Back-End systems and Salesforce, ensuring that the data required by employees stays up-to-date, and that our database is perfectly synced with any Salesforce entities that we need to keep track of.

With all these cards in our hand, we ought to determine each tool’s responsibility while establishing a suitable relation between them:

• Salesforce as an exception email sender.
• Gmail as an exception email receiver.
• The Salesforce Bridge as an exception emails reader and exceptions propagator.
• Sentry as an error tracking service.

In order to better realise the difference between the old and the new workflows, it is relevant to get an overview of the one we had in the past (note that the steps which will be explained are explicitly numbered in the incoming animated slides):

1. While the applications in Salesforce continuously run, an issue occurs during one of the numerous processes we use to handle data. Consequently, Salesforce gathers the stack trace before the exception arises and sends an email to the email group.
2. That same email is then forwarded to the Picnic developer email addresses we added to the email group.
3. Upon receipt, a notification is sent to a Picnic’s Slack channel.
4. Developers check their Slack messages and learn an issue occurred in Salesforce.
5. Developers access their email address inboxes and access the content of the email sent by Salesforce.

In the new workflow, we notably replaced Slack with the Salesforce Bridge and organised the course of action differently so as to have a more straightforward process. This time, let’s directly start with the old approach and see how we moved things around:

1. We removed Slack from the loop and replaced it with the Salesforce Bridge. The latter bears the responsibility of accessing the inbox containing the Salesforce exceptions and reads any unread emails periodically (we pull new emails every five minutes).
2. The bridge then propagates the stack trace contained in the email to Sentry. Note that we group issues based on the email content: issues with similar (yet slightly different) logs will end up under the same exception group, making investigations clear and effortless.
3. By applying the two previous points, not only did we restrict access to the email address inbox dedicated to receiving emails from Salesforce, but we also allowed developers to notice and seamlessly act on any issue that has occurred. Remember, Picnic developers already use Sentry every day, so we’re not adding another layer of laborious process; in fact, we’re improving it by centralising everything in one place.

In a real setup, considering the exception has already been propagated to Sentry, the process is straightforward:

1. We browse any issues which occurred over a predefined period of time (in this case, the past hour).
2. If no JIRA issue is linked, we create a ticket to highlight that the issue has been acknowledged and needs to be resolved.
3. Note that any similar subsequent events will be grouped under the same issue, allowing us to determine the time at which it started to appear along with the frequency.

Multi-organisation strategy

At the time of writing, Picnic keeps growing incredibly fast and as a consequence, we are already delivering groceries in three different countries: the Netherlands, Germany, and France. This implies we don’t only have one but three Salesforce organisations to manage (if you are curious and don’t know about our multi-org strategy, we write about it more extensively in the previous article here).

A couple of questions remain: How do we distinguish emails coming from one organisation to another, especially if they land in the same inbox? And how do we differentiate exceptions in Sentry?

These questions are intrinsically related, but let’s answer them in order. When using Gmail, there are different ways to modify the email address and still receive mail: either we add a ‘+’ sign combined with letters or numbers after the base of the email address, or we insert one or multiple dots anywhere in the email address.

In our case, we don’t need the latter because it doesn’t explicitly help us distinguish emails coming from an organisation, yet the former does. Considering the following email address “picnic@gmail.com”, if the organisation is associated with the Netherlands, we would request that Salesforce sends emails to e.g., “picnic+nl@gmail.com”: the recipient changes, although because of the ability to modify an email address without altering the main destination, the emails still end up in the same place.

In the same way Salesforce is organisation-based, all Picnic systems are market (NL, DE, FR) and environment (development, production)-based, meaning we have different instances of the same application. The Salesforce Bridge isn’t an exception and, as such, we mindfully open emails based on the recipient. Taking the previous example, for the NL instance of the Bridge in production, we would solely open emails that were sent to “picnic+nl.prod@gmail.com”, allowing us to propagate exceptions to Sentry that are market- and environment-specific.

Final note

As I mentioned earlier, the boundless journey to provide seamless and efficient services can only be pursued if we build the right tools, always scrutinise best practices, and continuously enhance operational processes from a business and a technical perspective.

By adjusting a single workflow, not only did we save developers a significant amount of time each day, but we also made issue-reporting unchallenging: we can track which release introduced a specific exception, how often it occurred and we can ensure it is followed up by linking a JIRA issue.

At Picnic, we endeavour to figure out the cleanest and most coherent solutions for any challenges we face. Is this a value that resonates with you? Would you love to tackle such problematics and handle end-to-end projects? Then look no further and have a peek at our job opportunities!