Why (and why not) use a Data Layer?

What’s a Data Layer?

Tag Management Systems can get data a variety of ways. For instance in DTM you can use query string parameters, meta tags, or cookie values- but in general, data for most variables comes from one of two sources:

  • To really take advantage of a tag management system like DTM, I may choose to scrape the DOM. I’m gonna call this the MacGyver approach. This uses the existing HTML and styles on a site to For instance, DTM could use CSS selectors to pull the values out of a <div> with the class of “breadcrumb”, and end up with a value like “electronics>televisions>wide-screen”. This relies on my site having a reliable CSS structure, and there being elements on the page that include the values we need for reporting.
  • If I want even more flexibility, control and predictability, I may work with developers to create a data layer. They would create a JavaScript object, such as “universal_variable.pageName”, and give it a value based on our reporting needs, like “electronics | televisions | wide-screen > product list”. This gives greater control and flexibility for reporting, but requires developers to create JavaScript objects on the pages.

Conceptually speaking, a data layer is page-specific (but tool-agnostic) metadata that describes the page and the actions a user may take on it. Practically speaking, a data layer typically consists of a JavaScript object that contains all of the values we’d want to report on for a given page or user.

Data layers are important because they save developers time by allowing them to abstract out the metadata into a tool-agnostic syntax that a TMS like DTM can then ingest and set as data elements. Whereas once I would have told IT “please set s.prop5 and s.eVar5 to the search term on a search results page, and set s.events to event20” now I can just say “please put the search term in a javascript object such as digitalData.page.onsiteSearchTerm and tell me what object it is.” Then the TMS administrators could easily map that to the right variables from there.

You can see an example data layer if you’d like, or you can pull open a developer console for this very blog and look at the object “digitalDataDDT” to see the data layer that is automatically created from Search Discovery’s wordpress plugin.

Why a Data Layer?

My friends at 33 Sticks also have a great blog post on the subject, but I’ll list out some of the reasons I prefer clients to use a Data Layer. To me, it’s an upfront investment for a scalable, easily maintained implementation going forward. It does mean more work upfront- you have to first design the data layer to make sure it covers your reporting requirements, then you’ll need developers to add that to your site. But beyond those upfront tasks, configuration in your TMS will be much simpler, and it will save you many hours of CSS guess work and DOM scraping, and it may prevent broken reporting down the line.

    Upfront LOE Maintenance LOE
Route Amount of Control Dev Analytics Dev Analytics
Old fashioned “page on code” Medium Heavy Heavy Heavy Heavy
DTM + “Macgyver” Low Minimal Heavy Minimal Heavy
DTM + Data Layer High Heavy Medium Minimal Minimal

Another potential benefit to a Data Layer is that more and more supplementary tools know how to use them now. For instance, Observepoint’s site scanning tool can now return data on not just your Analytics and Marketing beacons, but on your Data Layer as well. And one of my favorite debugging tools, Dataslayer, can return both your beacons and your data layer to your console, so if something is breaking down, you can tell if it’s a data layer issue or a TMS issue.

Ask Yourself

Below are some questions to ask yourself when considering using a data layer:

How often does the code on the site change? If the DOM/HTML of the site changes frequently, you don’t want to rely on CSS selectors. I’ve had many clients have reports randomly break, and after much debugging we realized the problem was the developers changed the code without knowing it would affect analytics. It’s easier to tell developers to put a data layer object on a page then leave it alone, than it is to tell them to not change their HTML/CSS.

How CSS-savvy is your TMS team? If you have someone on your team who is comfortable navigating a DOM using CSS, then you may be able to get away without a data layer a little more easily… but plan on that CSS-savvy resource spending a lot of time in your TMS.  I’ll admit, I enjoy DOM-scraping, and have spent a LOT of time doing it. But I recognize that while it seems like a simple short-term fix, it rarely simplifies things in the long run.

How many pages/page types are on the site? A very complicated site is hard to manage through CSS- you have to familiarize yourself with the DOM of every page type.

How are CSS styles laid out? Are they clean, systematic, and fairly permanent? Clearly, the cleaner the DOM, the easier it is to scrape it.

How often are new pages or new site functionality released? Sites that role out new microsites or site functionality frequently would need a CSS-savvy person setting up their DTM for every change. Alternatively, relying on a data layer requires a data-layer-savvy developer on any new pages/site/functionality. It is often easier to write a solid Data Layer tech spec for developers to reference for projects going forward than to figure out CSS selectors for every new site/page/functionality.

How much link-tracking/post-page-load tracking do you have on your site? If you do need to track a lot of user actions beyond just page loads, involving IT to make sure you are tracking the right things (instead of trying to scrape things out of the HTML) can be extremely valuable. See my post on ways to get around relying on CSS for event-based rules for more info on options.

What is the turn-around time for the developers? Many clients move to DTM specifically because they can’t work easily within their dev team to set up analytics. A development-driven data layer may take many months to set up, stage, QA, and publish. Then if changes are needed, the process starts again. It may be worth going through the lengthy process initially, but if changes are frequently needed in this implementation, you may find yourself relying more on the DOM.

Are there other analytics/marketing tag vendors that may use a data layer? You may be able to hit two birds with one stone by creating a data layer that multiple tools can use.

Have you previously used another tag management system? Often, a data layer set up for a different tool can be used by DTM. Similarly, if the client ever moves away from DTM, their data layer can travel with them.

Does the site have jQuery? The jQuery library has many methods that help with CSS selectors (such as .parent, .child, .closest, .is, .closest…). A CSS-selector-based implementation may be more difficult without jQuery or a similar javascript library.

Who should create my Data Layer?

Ideally, your data layer should be created by your IT/developers… or at bare minimum, developers should be heavily involved. They may be able to hook into existing data in your CMS (for instance, if you use Adobe Experience Manager you can use the Context Hub as the basis for your data layer), or they may already have ideas for how they want to deploy. Your data layer should not be specific to just your Analytics solution; it should be seen as the basis of all things having to do with “data” on your site.

Yet frequently, for lack of IT investment, the analytics team will end up defining the data layer and dictating it to IT. These days, that’s what most Tech Specs consist of: instructions to developers on how to build a data layer. Usually, external documentation on data layers (like from consulting agencies) will be based on the W3C standard.

The W3C (with a task force including folks from Adobe, Ensighten, Microsoft, IBM…) has introduced a tool-agnostic data layer standard that can be used by many tools and vendors. The specifications for this can be found on the W3C site, and many resources exist already with examples. Adobe Consulting often proposes using the W3C as a starting point, if you don’t have any other plans. However, in my experience, generally that W3C is just a starting point. Some people don’t like the way the W3C is designed and most everyone needs to add on to it. For example, folks might ask:

  • why is “onsiteSearchTerms” part of digitalData.page? Can I put it instead in something I made up, like digitalData.search?
  • I want to track “planType”- the W3C didn’t plan for that, so can I just put it somewhere logical like digitalData.transaction?
  • I don’t need “digitalData.product” to be in an array- can I just make that a simple object.

The answer is: yes. You can tweak that standard to your heart’s delight. Just please, PLEASE, document it, and be aware that some tools will be built with the official standard in mind.

The Phased Approach

Many folks adopt a TMS specifically because they don’t want to have to go through IT release cycles to make changes to their implementation. You can still use a TMS to get a lot of what you need for reporting without a data layer and without a ton of CSS work. It may be worthwhile to put a “bare minimum” TMS deployment on your site to start getting the out of the box reports and any reports that don’t require a data layer (like something based on a plugin such as getTimeParting), then to fill in the data layer as you are able. I’d be wary though, because sometimes once that “bare minimum” reporting is in place, it can be easy to be complacent and lose some of the urgency behind getting a thorough solution implemented correctly from the start.

Conclusion

I fully understand that a properly designed data layer is a lot of work, but in my experience, there is going to be a lot of effort with or without a data layer- you can choose for that effort to be upfront in the planning and initial implementation, or you can plan on more longterm maintenance.

Should I have one DTM property, or many?

It’s common for a company in DTM to have multiple sites/groups/regions they want to deploy DTM on. One of the more important decisions you can make, early on, is whether to keep those different sites as separate properties, or keep them in a single DTM property. I know future iterations of the DTM tool will make up for some of the current issues we run up against when using multiple properties, but for now my advice is to keep as few properties as possible- it’s so much easier to maintain.

Note: Your properties in DTM are NOT related to your Adobe Analytics login companies or Report Suites- one property can go to many report suites (with custom logic in the s_code) or many properties can feed into the same report suite. So instead, you should consider the following:

How similar are the implementations?

Will the sites all have the same source for data, like a data layer, or CSS selectors? Do they use the same general analytics variable map (eg, “eVar15 is always search terms”)? Will they generally use the same third party tags? Do they all use Adobe Target or other DTM tools?

If your sites are similar and you want to keep separate properties, you can in theory set up one property and get it configured just right, then copy your rules and data elements over to all the other properties. But be warned: after rules are copied, if you need to update that rule, you must update all its copies too. To be honest, this can be beastly to maintain. I’ve seen it work, but only if folks use documentation to note what changes have happened where.
Imagine you want to add a new variable to your Search Results page, and you have 6 properties that have similar Search Results rules- you’d need to make that change in all 6 properties, and approve and publish in 6 places.
If, however, of your 6 sites, only 3 have a search results page, and all of them are pulling the search term out of a different query string parameter- then you’re going to be maintaining 6 separate rules anyways, and there is no benefit to keeping one property.

Another big consideration are the tools for each site. Tools in DTM are generally intended to be deployed on ALL pages using that DTM property- meaning you can’t turn on Target or the Marketing Cloud service for just one portion of a property. For Adobe Analytics, you can suppress the tool on certain pages, but in general, plan on Tools in DTM running on every page, without being able to suppress them for certain pages.

How similar are the publication timelines?

Will one site go live with the Analytics tool sooner than another? Does another site have a lengthy QA process?  Would it be ok to publish changes in DTM that will push out to all sites at once?

If Site B will have their old Analytics implementation removed from their site and ready for DTM to take over in June, and Site B won’t be ready until August, then you’d need to plan things very carefully in DTM. If the plan to transition to DTM really varies between your sites, then you may want to keep separate properties, at least for now. Perhaps in the future you could merge them back together.

Who will be doing the DTM work for each site?

Will the person implementing/administering DTM for one site be the same as for another site? Are you concerned about someone adding third party tags for site B having access to rules that could apply to site A? 

Within a company, you can use groups to restrict Users to certain properties. So you could say “Joe should only have access to Site B” if Site B is its own property.

Within a property, you can use groups to restrict Users to certain tools (for instance, you can give someone access to Third Party Tags but not to Adobe Analytics). You cannot, however, restrict certain rules- you can’t say “Joe can edit the homepage rule, but not purchase confirmation”.

On a side note, you can not restrict Admin access- if someone needs to make changes to tools, manage users, or view embed codes, that someone will have admin access for your whole company. So you can’t say “Joe should have access to the embed options for Site A but not Site B”.

Conclusion

If your sites are at all similar in their implementation, I think it’s worth it to try to make a single property work, but I definitely understand there are situations in which you must keep separate properties. Hopefully this post will help you decide on the best possible arrangement for your company.

What the DTM “top down” approach means for your page performance

Any javascript framework, including all Tag Management Systems like DTM, have the potential to ADD more javascript weight to your page. But if you approach things the right way, this javascript weight  and its effect on your page performance can by mitigated by using DTM to optimize how your tools and tags are delivered. In a partner post, I’ll be talking about how to get the most out of DTM as far as Third Party Tags go, but I think one key concept is worth discussing explicitly.
You may have heard DTM be referred to as a “Top Down” TMS. For instance, this appears in some of the marketing slide decks:
topdown

While yes, it’s worth discussing this as a holistic approach to your digital analytics, it actually has a very real effect on how you set your rules up and how that affects page performance. That’s what I hope to discuss in this post.

In a different TMS, or even in DTM if I haven’t changed my mindset yet, I may be tempted to do something like this:

down-upRules

Where I have differing rules for different scopes as well as for different tags (we’re pretending here that “Wuggly” is a Third Party Marketing Pixel vendor).

DTM does what it can to defer code or make it non-blocking, but there are parts of the DTM library which will run as syncronous code on all pages. Some of that is because of the way the code needs to work- the Marketing Cloud ID service must run before the other Adobe tools; older Target mbox code versions need to run syncronously at top of page. But there is also the code in the library that serves as a map for when and how all of the deferred code should run. For instance, my library may include the following:

downUpCode

All of this logic exists to say “if the current pageType is “home page”, run this code non-sequentially”.  The name, conditions and event code for each rule run on each page as part of the overall library- these serve as a map for DTM to know which code must run, and which code it can ignore and not run.
You’ll notice the code for the two rules is completely identical, except for the rule name (in blue) and the source of the external script (in yellow). Most importantly, the conditions (in green) are identical. Whereas if they shared a rule, we might see the exact same thing as above accomplished with half as much code:

topDownCode

I now have ONE rule, which would be used for ALL logic that should run on the Home Page. The part of the library that runs on every page to check against conditions only has to check if the “pageType” is “home page” once, rather than twice. And DTM still loads the two scripts as separate non-sequential javascript. This doesn’t look like a major change, but when viewed across a whole implementation, where there may be dozens of rules, it can make a big difference in how many rules and conditions DTM must check on every page.

In the DTM interface, this would look like this:
topDownRules

If I want to know which rules contain my “Wuggly” codes, I can use the  “Tag Name” filter in the rules list, which will show me all rules that have a third party tag that includes “Wuggly”:
topDownFilter

This is filtering based on the Tag Name specified when you add the tag code:
topDownTagName

Using this approach, where your rules are based on their scope (or condition) and contain all logic- Analytics, Target, third party- that applies to that scope can not only lighten your library weight, but it can also keep your DTM implementation organized and scalable, but it may also require a change of mindset for how DTM is organized- if someone else needed to deploy a tag on Product Detail Pages, they wouldn’t need to create a rule, but rather, they could see a “Product Detail Page” rule already exists with the scope they need, and they need only add the third party tag.

There is one potential downside to consider, though- the approval and publication flow is based on Rules in their entirety. You can’t say “publish my changes to the analytics portion of this tool, but not to the third party tag section”. Still, if you are aware of this as you plan the publication of new features, this potential drawback rarely overrides the advantages.

Get the most out of DTM for deploying Third Party Tracking

One of the benefits of using a tag management system like DTM is the ability to lighten the load on your page by moving tracking pixels into DTM. Now, simply moving code into DTM may not improve page performance- there are best practices you need to follow to get the most out of what DTM can offers.

1. Decide on the scope

When the DTM library loads, it defers as much code as possible to later in the page. In order to map out what should run where it must run through each of your rule conditions and see which conditions are currently met. That means that additional rules and additional conditions will actually slow down the synchronous part of your DTM library. When possible, don’t create a new rule for each new tag, but rather, have rules be specific to their condition. I have a partner post about how to improve page performance when planning out your rules, but for now, try to start thinking of your rules in terms of the user action- have one rule for when the user sees a product details page, for example, rather than a series of Product Details Page rules, each with a different tag.

2. Decide which type of DTM script to use

Since 3rd party Tag vendors generally deliver their code in HTML form, intended to be pasted directly into your page, there are usually a few changes you need to make before DTM can fire the code non-sequentially.  What you do varies by the tag.
Before proceeding, you need to decide: should you use non-sequential HTML or non-sequential javascript? DTM loads non-sequential HTML by setting it in a side <iframe> so it can load the content without blocking anything else. This can work well, but has some downsides: that iframe can’t get all the same information the parent page can. This includes many data elements- for security reasons, it can’t reference a data element that pulls from Custom JS or JS objects. If you’re referencing a data element, you need to use “%dataElementName%” rather than “_satellite.getVar(“dataElementName”)”. This isn’t the most supported usage, so definitely test it thoroughly.

3. Remove unneeded pieces

Next, remove any <noscript> tags: they won’t do any good in a Javascript-based framework like DTM. These days, these tags aren’t really needed and may actually inflate your data because the only folks who don’t have JavaScript enabled are bots.

4. Convert the code to suit your script type

Next, convert the code to the appropriate format (see below) and add it in a rule as a Third Party Tag.

Purely Script

Some tags are already just javascript. For instance, take this code from facebook:

<!-- Facebook Pixel Code -->
<script>
!function(f,b,e,v,n,t,s){if(f.fbq)return;n=f.fbq=function(){n.callMethod?n.callMethod.apply(n,arguments):n.queue.push(arguments)};if(!f._fbq)f._fbq=n;
n.push=n;n.loaded=!0;n.version='2.0';n.queue=[];t=b.createElement(e);t.async=!0;
t.src=v;s=b.getElementsByTagName(e)[0];s.parentNode.insertBefore(t,s)}(window,
document,'script','//connect.facebook.net/en_US/fbevents.js');

fbq('init', '123456789123');
fbq('track', 'PageView');
</script>

<noscript>
<img height=""1"" width=""1"" style=""display:none""
src=""https://www.facebook.com/tr?id=1552630471664132&ev=PageView&noscript=1""
/></noscript>

<!-- End Facebook Pixel Code -->

I would remove the <script> tags, the <noscript> portion, and HTML comments, and paste it directly as JavaScript:

!function(f,b,e,v,n,t,s){if(f.fbq)return;n=f.fbq=function(){n.callMethod?
 n.callMethod.apply(n,arguments):n.queue.push(arguments)};if(!f._fbq)f._fbq=n;
 n.push=n;n.loaded=!0;n.version='2.0';n.queue=[];t=b.createElement(e);t.async=!0;
 t.src=v;s=b.getElementsByTagName(e)[0];s.parentNode.insertBefore(t,s)}(window,
 document,'script','//connect.facebook.net/en_US/fbevents.js');

fbq('init', '123456789123');
fbq('track', 'PageView');

I know Facebook and Chango are two tag types that fall in this category.

Simple Pixel

The easiest of pixels are those that just require the loading of a tiny invisible image. For instance, let’s say I got the following Yahoo dot code:

<img src="https://sp.analytics.yahoo.com/spp.pl?a=123456789&.yp=98765&js=no" height="0" >

If I want to load an image like this asynchronously, I could paste it in, unchanged, as a non-sequential HTML third party tag. As mentioned earlier, this would create an iframe that loads on the side of the page, so as to not slow down page performance.

I could also add it as a non-sequential javascript third party tag using document.body.appendChild to append it to the body, whether it has finished loading or not. This also makes it so you can add these pixels in event-based rules on SPAs or post-page-load user actions.

var dcIMG = document.createElement('img');
dcIMG.setAttribute('src', 'https://sp.analytics.yahoo.com/spp.pl?a=123456789&.yp=98765&js=no');
dcIMG.setAttribute('height','1');
dcIMG.setAttribute('width','1');
dcIMG.setAttribute('border','0');
dcIMG.setAttribute('style','display:none');
document.body.appendChild(dcIMG);

To my knowledge, Yahoo Dot, Bing, Vibrant/Intellitxt, Gumgum are examples of simple pixel code vendors.

Pixel with query params

Some vendors have simple pixels, with query parameters in the src url that helps tell the vendor what they need to know. (Coincidentally, this approach is the approach used by Adobe Analytics.) Let’s say I got this code from Doubleclick:

<!--<script type="text/javascript">
var axel = Math.random() + "";
var a = axel * 10000000000000;
document.write('<iframe src="//0.fls.doubleclick.net/activityi;src=123456789;type=clientsale;qty=1;cost=[Revenue];ord=[OrderID]?" width="1" height="1" frameborder="0" style="display:none"></iframe>');
</script>

<noscript>
<iframe src="//0.fls.doubleclick.net/activityi;src=123456789;type=clientsale;qty=1;cost=[Revenue];ord=[OrderID]?" width="1" height="1" frameborder="0" style="display:none"></iframe>
</noscript>
<!-- End of DoubleClick Floodlight Tag: Please do not remove -->

I could add this to my site like this, using data elements in place of [Revenue] and [OrderID]:

var axel = Math.random() + "";
var a = axel * 10000000000000;

var dcIMG = document.createElement('iframe');

dcIMG.setAttribute('src', "//0.fls.doubleclick.net/activityi;src=123456789;type=clientsale;qty=1;cost="+ _satellite.getVar('purchase: revenue') +";ord="+ _satellite.getVar('purchase: order id') +"?");


dcIMG.setAttribute('height','1');
dcIMG.setAttribute('width','1');
dcIMG.setAttribute('Border','0');
dcIMG.setAttribute('style','display:none');
document.body.appendChild(dcIMG);

Pixels with Script tags

Many vendors require you to add their javascript file to your site, as well as set some variables.

For instance, I might get this code from twitter:

<!-- Twitter single-event website tag code -->

<script src="//platform.twitter.com/oct.js" type="text/javascript"></script>

<script type="text/javascript">twttr.conversion.trackPid('123456', { tw_sale_amount: 0, tw_order_quantity: 0 });</script>

<noscript>

<img height="1" width="1" style="display:none;" alt="" src="https://analytics.twitter.com/i/adsct?txn_id=l5jl4&p_id=Twitter&tw_sale_amount=0&tw_order_quantity=0" />

<img height="1" width="1" style="display:none;" alt="" src="//t.co/i/adsct?txn_id=l5jl4&p_id=Twitter&tw_sale_amount=0&tw_order_quantity=0" />

</noscript>

<!-- End Twitter single-event website tag code -->

So there is a script that needs to run, then script that needs to fire afterwards. We can remove the noscript portions, but we still need to figure out a way to get the .js script to run before I fire twitter.conversion. There aren’t a lot of options for running a .js file after a page has loaded- if you have jQuery, you can use $.ajax, but if you don’t have jQuery you can use the below to append the .js file, make sure it has completed, then run twttr.conversion:

var dcJS = document.createElement('SCRIPT');
var done = false;

dcJS.setAttribute('src', '//platform.twitter.com/oct.js');
dcJS.setAttribute('type','text/javascript');

document.body.appendChild(dcJS);
dcJS.onload = dcJS.onreadystatechange = function () {
     if(!done && (!this.readyState || this.readyState === "loaded" || this.readyState === "complete")) {
          done = true;
          callback();
 
           // Handle memory leak in IE
           dcJS.onload = dcJS.onreadystatechange = null;
           document.body.removeChild(dcJS);
     }
 };
 function callback(){
      if(done){
           twttr.conversion.trackPid(pid,{
                tw_sale_amount:_satellite.getVar('purchase: total revenue'), 
                tw_order_quantity:_satellite.getVar('purchase: total units') 
           }) 
      }
 }

Google Adwords/Remarketing, Twitter, Linkedin, and Eloqua can all fit this general idea.

5. Validate the tag

There are many tools on the internet for validating tags are working. Some vendors, like Google, may have their own tool, but most tools can be validated either within a Developer Console or by using tool-agnostic tools like Ghostery or the chrome Observepoint plugin. Most methods require opening a developer console. An easy way to do this in most browsers is to right-click anywhere on the page and select the “Inspect” option. For instance, in Chrome:

chromeConsole

First, check there are no errors in the console. Then, to validate a specific tag:

Using the Observepoint plugin: Within the developer console, go to the “Observepoint” tab:

observepoint

Click the “Recording” button, then refresh the page. Observepoint should show every tracking technology running on the page, including the one you are validating.

Using the built-in tools in your browser: Open the network tab, then refresh the page. Often you can search for the name of the tag- for instance, here I’m searching for “twitter”, and it shows me that data was sent to twitter:

netTab

Other vendors use slightly more disguised names. Check with the vendor if you need more details on how to validate.

Find the total number of values in an Adobe Analytics Variable Report

“How do I know how many rows/unique values are in my eVar report?”
I hear this question fairly often, and since the answer isn’t currently extremely straight-forward, I figured I could walk through the solution here. So let’s say you have a report with many pages:
reportPagination
Rather than trying to get to the last page of the report somehow to see how many row numbers there are, you can find out about the number of unique values in the report using the Row Count calculated metric.
To set this up, go to add a metric to the report, then “Add” to get to the Calculated Metric wizard:
addCalcuatedMetric

Give the metric an intuitive name, like “Report Rows” or “Unique Reported Values”, then search in the formulas for “Row Count”:
rowCount

Drag “Row Count” over to the Calculated Metric Definitions, and save. That’s all there is to it!

RowCountCalc

Now you can have that as a metric in your report. For whatever reason, it doesn’t always show nicely if it is the ONLY metric selected, so I recommend having at least one metric (like visits) alongside it as well. It doesn’t make a ton of sense in the report, line-by-line: every line will show the same number, and that number represents the aggregate number of values in the report. So in the below, I can see this particular report has 140 unique rows/values in it:
RowCountReport

I do believe in the future, a simpler way of doing this may be a bit more built in, but for now, here is a relatively easy way to get a count of values for a report!

How to approach Product Finding Methods

Product Finding Method (PFM) is a valuable custom report used by online retailers to tie Cart Adds and eventual Revenue to how users found the product. Unfortunately, these reports are often poorly configured or misunderstood. It is one of the best use cases for using the Merchandising setting on an eVar, but Merchandising, as a concept, is one of the more complicated topics in implementation.

Primer on Merchandising

In order to tie that PFM value to an individual product, and not the overall visit or visitor, we need to use Merchandising. And since we may not know the eventual product at the time that we know the finding method (for instance, if a user is using internal search, at the time we know they are searching, we don’t know what product they’ll eventually land on), we need to use Conversion Syntax- meaning we set it like a typical eVar (instead of in the products string) but have to configure at what point that eVar should be tied (or “bound”) to a product. Take the following user experience into consideration, where eVar5 is used for Product Finding Methods:

PFMflow

In the above, if I did NOT use Merchandising, my report might look like this, where External Campaign gets credit for everything up until it gets overwritten by Internal Search– at which point Internal Search gets credit for everything, including the entire Purchase event.

PFMnoMerchReportIf I DO use Merchandising, the External Campaign gets credit for whatever happens to the product it was bound to- in this case, the Blue Wug. Then Internal Search gets bound to the Red Wug, and gets credit for whatever happens to the Red Wug product:

PFMwMerchReport

We have to be very careful with the binding events (the event that tells Adobe to take whatever value we currently have floating around for that eVar and stick it to the current product), though- merchandising eVars won’t get credit for anything that happens if they are not bound to any product, or if that product isn’t currently present. For instance, in the example above, if you pulled in Internal Searches as a metric in that report, you’d see no PFMs were getting credit for the Internal Search event- even though eVar5 is set on the same page as our search event. That’s because that “Internal Search” value for eVar5 is waiting until a “binding event” to attach it to a product before it will get credit for any metrics, and the “External Campaign” value for eVar5 only gets visibility into things that happen to the product it’s bound to (Blue Wug). No products on the Internal Search page means Conversion-Syntax Merchandising eVars get no credit for the Search event.

How to Plan Your Product Finding Methods

To design your PFM solution, start by thinking of the different ways a client can add an item to their cart. We’ll call these “add to cart location” and give that its own eVar. Typical scenarios are:

  • product details page
  • quick view
  • from wishlist
  • from order history
  • recommendation/cross sell modules

Next, figure out all the ways a user could get to any of those above scenarios. These are Product Finding Methods and should get their own eVar. Typical PFMs include:

  • internal search
  • external site (referrerss and natural search)
  • external campaigns (that take the user directly to the PDP)
  • internal promotion (eg, homepage hero banner)
  • browse (naturally navigating through the menu or breadcrumbs)
  • cross-sell
  • recommendations
  • wishlist
  • order history 

You’ll notice some of your cart add locations may also be product finding methods- that’s ok. Note that Products Detail Page is NOT a PFM- the Product Details page is specific to the product, which is the thing being found not the thing doing the finding.

Both of these eVars will need to be set up for Merchandising (this is set in the Admin console>Report Suites>Edit Settings>Conversion>Conversion Variables). For the Add to Cart Location eVar, since we know the product at the time we know the add to cart location, you can use “product syntax”.

eVar4PFM

This is preferable because it leaves little room for error and has the easiest configuration. Let’s say I’ve set eVar4 to be my “Add to Cart location” eVar. At the time my user adds any item to their cart, I would set eVar4 (often, you can just set it to the current page’s page type):

s.products=";sku1;;;;eVar4=PDP"
s.events="scAdd"

But for Product Finding Method, you often don’t know what product the value for our eVar will eventually bind to, so we need to set it to conversion syntax, and we need to carefully consider which events should bind the current PFM value to the current product. Generally, Product View (ideally a custom event, not prodView) and Cart Add suffice as binding events.

eVar5pfm

DO NOT set to bind on ALL events, as many events happen on your site where your PFM eVar doesn’t have a desirable value.

Next, to implement, we need to figure out where to set each of our PFM values. If you’re using a tag management system like DTM, you can often just piggy back on existing rules for many of your finding methods, but others may need to live in global logic (like in your s_code). See below for a possible configuration.

internal search Set on internal search results page
external site Set in global logic- if no other PFM is present, and the user came from a domain other than your own
external campaigns Set in global logic, if s.campaign is set
internal promotion Set in global logic, if internal campaign eVar is set (or internal campaign query string parameter is present).
browse Set on all Product List Pages and Category pages
cross-sell Set when user clicks or clicks through on a cross-sell module
recommendations Set when user clicks or clicks through on a recommendation module
wishlist Set on Wish List page
order history Set on Order History page

Some folks manage PFM entirely in their s_code or in their (global DTM tool settings) based on whether other variables have been set (“if internal search term eVar has a value, set eVar5 to ‘internal search”‘). For instance, one client has something similar to this, in addition to setting their PFM eVar directly on certain pages (like their List Pages and Category pages, or certain microsites):

//easy way to get just the hostname of the referring site
   var a=document.createElement('a');
   a.href=document.referrer;
   s.refDomain=a.hostname;

if(!s.eVar5){ //if PFM not currently set on the page
   if(s.campaign){s.eVar5="campaign"} 
   else if(s.eVar10){s.eVar5="internal search"}
   else if(s.eVar23){s.eVar5="recommended product"} 
   else if(s.pageName=="My Account>Wishlist"){s.eVar5="my wishlist"}
   else if(document.referrer==""){s.eVar5="direct or bookmarked"}
   else if(s.refDomain.indexOf("mydomain.com")==-1){s.eVar5="external site"} 
 }
   else if(s.eVar15){s.eVar26="internal campaign"} 
}

Conclusion

Hopefully, this gives some ideas and examples for how to get valuable reporting on how users are finding and purchasing products on your site. I’d love to hear about potential scenarios or solutions I’ve missed!

An alternative to using CSS for Event-Based Rules

A lot of the magic that happens in DTM is based on CSS (cascading style sheets) and CSS Selectors (using the styles applied to HTML elements as a map to your site). Even if you have a data layer to avoid relying on CSS for your data elements, Event-based Rules still rely on CSS Selectors to figure out which elements to listen to. For instance, if I want DTM to listen for clicks on my blog menu, which looks something like this:

<li id="menu-item-82" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-82">
     <a href="http://digitaldatatactics.com/beaconParser/">Beacon Parser Tool</a>
</li> 

I might tell it to listen to link/anchor tags (“a” tags) that are within list items (“li” tags) that have a class of “menu-item”.  The CSS Selector for this would be “li.menu-item a”, so I might create this rule in DTM:

ebrCSS

This should work, BUT there are a few potential problems:

  • it relies on the CSS of my site staying the same, and on my ability to interpret CSS selectors to get one with the right level of granularity.
  • Developers wouldn’t know that Analytics is relying on that “menu-item” class, and they might change it without notice.
  • An implementation may have dozens of these Event-based rules, which could require a lot of time to set up and test.
  • Lastly, if I want to pass a value like “nav click:beacon parser” I have to figure out a way to dynamically grab the value “beacon parser” from the surrounding HTML.

There is an alternative: instead of relying on classes, we could work with developers to add another attribute to the elements we want to listen to. This attribute would be completely arbitrary- I’m going to use “data-track” for this post. So I might ask developers to add “data-track=’nav click'” to that element:

<li id="menu-item-82" class="menu-item menu-item-type-custom menu-item-object-custom menu-item-82" data-track="nav click:beacon parser">
     <a href="http://digitaldatatactics.com/beaconParser/">Beacon Parser Tool</a>
</li>

Then in DTM, I could set up a rule that listens not to CSS, but for clicks on any item that has “data-track” on it:
ebrDatatrack

Or if I wanted, I could have this particular rule only fire on clicks where “data-track” started with “nav click”: “[data-track^=’nav click’]”.

This approach has some benefits:

  • Developers aren’t likely to accidentally change the ‘data-track’ attribute
  • Setting up rules in DTM doesn’t require deep knowledge of CSS
  • You don’t have to rely on the HTML around the element to provide dynamic values (like “beacon parser” in my example above.) Whatever values I want in my reporting, I can have put in friendly wording into my data-track attribute.

As well as a few potential downsides:

  • I’d need to get my developers to add this attribute to all elements I want to track.
  • I’d need some sort of management on the values of those attributes (I may want to document that we’re using “nav click” and keep some standards and consistency across how data-track is used across the site.)

Ultimately, for many clients, the benefits of having control and flexibility far outweigh the upfront work of adding those tags.

Hopefully this approach can provide a solution to some folks out there, or at least can serve as a starting point for discussing the different options for setting event-based rules. I’d love to hear about folks who have used this solution or other solutions to keep CSS selector dependency to a minimum!

How do I use DTM for a Single Page App?

The question of how to use DTM on Single Page Apps (SPAs) is a VERY hot item right now. By Single Page App, I’m referring to a full user flow contained on a single web page, so as to provide the user a more seamless experience. Often, these pages act like typical web pages, but they don’t always change URLs or load new resources. Many common web development technologies, such as Angular.js, Ember.js, and AJAX use SPA principles.

Unfortunately, there isn’t a single great answer for how to deploy DTM- it depends on many things. I’ll work through some of the options and the limitations to be aware of.

Suppressing Page View Beacons

Whatever method you take for tracking page views in a SPA, keep in mind most SPAs do have one true “page view” when the DOM first loads. If you are going strictly with the DCR or EBR route, you may need to suppress the initial page view beacon the Analytics tool will want to set by default. Otherwise, in the example below where the developers are firing a direct call rule on all page views, you’d get TWO beacons on the first page and 1 on all subsequent pages.

Picture1

Data Layer Considerations

You’ll need to make sure that whatever the sources of your Data Elements are (CSS selector, javascript objects, cookies…) have the correct values BEFORE your rule is triggered. I have an example site showing one way you might do this for a data layer (though you’ll need to look in the source code), but ultimately it’s going to depend on your site.

Variable Persistence

One last consideration is that once your Analytics object exists (as in, the “s” in “s.pageName”), variables set on it will continue to exist unless specifically overwritten. In most cases, you’d overwrite s.pageName with a new value so it isn’t a problem, but something like s.eVar5 may be set on the first beacon in your SPA, and not desired on subsequent beacons. You can use s.clearVars() to “refresh” your “s” object, but you have to make sure it fires at the right time- for example, after the beacon on Page A, but before DTM starts mapping Data Elements to variables for the beacon on Page B. How you do this will depend on the overall deployment method you choose.

Deployment Methods

1) Direct Call Rules

Perhaps the most straight-forward approach is to have developers fire a Direct Call Rule, like _satellite.track(“page view”) on every thing YOU consider a page view, whether it’s a fresh new DOM or not.

Advantages: Disadvantages: 
  • You have ultimate control over when a page view in considered a page view.
  • If you need to clear out variables between beacons (for instance, you set s.eVar5 in the first beacon in the SPA, and don’t want it in the second beacon), Direct Call Rules don’t provide a great place to use something like s.clearsVars(). There are some potential work-arounds, but none are ideal.
  • Developers need to add more DTM-specific code (satellite.track) to your pages.
  • Direct Call Rules don’t allow for extra conditions (like “fire THIS logic on pageA, and THAT logic on pageB”) in the interface.
  • Direct Call Rules don’t “stack”- if multiple rules have conditions that are met, multiple rules will fire.

2) pushState or hashChange

Many SPA frameworks, like Angular, use a certain flag to let the browser know the user is viewing a new “page”.

pushState

DTM can listen for this flag in an Event Based Rule using a pushState or hashChange condition.

ADVANTAGES: DISADVANTAGES: 
  • No additional code is needed- most SPA frameworks are already firing something DTM can listen to
  • It’s an Event Based Rule, which allows you to fire clearVars(), and set extra conditions
  • Because you are listening for an event set by the framework, you have less control over timing. Updating a data layer BEFORE the “pushState” event is detected would be critical.
  • Event Based Rules don’t “stack”- if multiple rules have conditions that are met, multiple rules will fire.

3) Custom Event EBR

Another option, which feels a bit like a blend of the first two options, is to use a Custom Event-based Event Based Rule (and no, that’s not a typo- it’s an EBR based on the JavaScript Concept of a Custom Event). It’s possible Developers are already using this Custom Event concept for their own code and purposes, and DTM can just listen for it… or you can have developers set one specific to our DTM needs by using something like my digitalData.userAction hack.

ADVANTAGES: DISADVANTAGES: 
  • You have a little more control over timing
  • It’s an Event Based Rule, which allows you to fire clearVars(), and set extra conditions
  • May require more developer work- similar level of effort as DCRs
  • Event Based Rules don’t “stack”- if multiple rules have conditions that are met, multiple rules will fire.

 4) (NEW OPTION!) “dataelementchanged” Event Based Rule

Just in the last few weeks, a new option emerged as part of the AEM Context Hub integration. This allows DTM to listen for changes to certain data elements- for instance, you could have a rule fire whenever the “pageName” has changed. My friends at 33 sticks have a great blog post about this already for more info.

ADVANTAGES: DISADVANTAGES: 
  • You have a little more control over timing
  • It’s an Event Based Rule, which allows you to fire clearVars(), and set extra conditions
  • Requires careful consideration of when the data layer changes/loads
  • Event Based Rules don’t “stack”- if multiple rules have conditions that are met, multiple rules will fire.

Setting up an Event-based Rule that be fired directly like a Direct Call Rule

Update:
As proud as I am of this solution/workaround, it may be far less needed now that you can use the dataelementchanged  condition to fire an Event Based Rule. Instead of using custom events, you can just have DTM listen for when your pageName Data Element has changed. 

The current limitations

If developers want to fire a DTM rule directly from their code (say, they want to make sure a beacon fires only after their data layer is ready), typically they would fire a Direct Call Rule, with its very specific syntax: _satellite.track(“rule string here”). There are, however, some limitations to this method.

Direct Call Rules:

  • Don’t allow for multiple conditions (you can’t say “if _satellite.track(“cart add”) is fired AND the current page has “/products” in the URL“)
  • Don’t allow for multiple arguments (you can’t pass _satellite.track(“cart add”,”sku1″) to attach the added SKU to the rule)
  • Don’t allow for firing s.clearVars() before your rule sets up your analytics variables (to clear out variables from previous beacons on the same DOM).
  • Require very specific syntax- they MUST be “_satellite.track()”

And unfortunately, both Direct Call Rules and Event-based rules don’t “stack”- if a certain condition triggers multiple similar rules, each rule will fire its own beacon. This is different from Page Load Rules, where if multiple rules have conditions being met by the current page, they all wrap nicely into a single page view beacon.

An alternative

To get around some (but maybe not all) of these limitations, I’ve been playing with another possible option, where we use the Custom Event conditions of an Event Based Rule to accomplish nearly the same thing. After getting it set up, I can fire something like this:

digitalData.userAction("cart add","sku1")

…to fire an Event-Based Rule in place of a Direct Call Rule. There are a few things I need to do in DTM to make this digitalData.userAction work.

Set Up the Logic

First, I have to set up the logic in a Page Load Rule- set to fire on DOMReady (no need for it to be sooner)- that will merely hold the following as a Sequential Javascript Third Party Tag:

//make sure digitalData is defined to prevent errors
if(typeof digitalData=="undefined"){
  digitalData={}
}

//create fake DOM item to bind the event to
var fakeDiv = document.createElement('div');
fakeDiv.setAttribute('id', "dtmHolder");
fakeDiv.setAttribute('height','1');
fakeDiv.setAttribute('width','1');
fakeDiv.setAttribute('style','display:none');
document.body.appendChild(fakeDiv);

//define custom event
digitalData.userAction=function(n,details){

 document.getElementById("dtmHolder").addEventListener("dtmEvent", function(e) {
    console.info("Event is: ", e);
  })

  // First create the event
  var dtmEvent = new CustomEvent(n, {
    detail:""
  });

  jQuery("#dtmHolder").attr("detail",details)
  
  // Trigger it!
  document.getElementById("dtmHolder").dispatchEvent(dtmEvent); 
}

(Update: note that this code should not run before the DOM is created- it will create an error if you try to run it at page top because you are trying to append something to a body that doesn’t exist yet).
Now, whenever a developer fires digitalData.userAction(“string here), you can listen for that string as the Triggered Event Type in a Custom Event Event Based Rule. Obviously, you can alter the above code if you want a function named something other than digitalData.userAction.

Set Up an Event Based Rule

The rule will need to be bound to the CSs selector of the tiny fake div (“#dtmHolder”) we created for the custom event to bind to:

2016-04-20_12-38-12

You can create as many of these rules as you want, for whatever different strings you pass into digitalData.userAction()where the “triggered event type” reflects that string.

Pass Additional Info

If you want to pass a second argument ( e.g. digitalData.userAction(“cart add”,”sku1″)) I currently have that second argument  passing as a new attribute (“detail”) on the tiny invisible div, so you can access it off the “this” object directly in the rule:2016-04-20_12-41-52

You can give this a try at my ugly test site– open a developer console, turn on DTM debugging, and fire either digitalData.userAction(“cartAdd”,”sku123″) or digitalData.userAction(“pageView”) to see two example rules at work.

Run ClearVars

This opens the ability to run s.clearVars on s.t() beacons in cases where multiple beacons may be firing on a single DOM. (As a reminder, if you’re using the old DCR route, there are some hack-ish options for doing this- we call it daisy-chaining).

In an Event Based Rule, there IS a code block that runs before the Analytics tool, giving you a perfect opportunity to make sure you are starting with a ‘clean slate’ of variables: the Conditions Custom Code block. Just create a new Rule Condition with “custom” criteria, then put “s.clearVars()” in the code block, followed by “return true” (so that DTM doesn’t think some condition didn’t pass):

2016-04-20_12-52-49

You can also apply additional conditions, like “only fire this “cart add” rule on certain pages”, by adding more criteria under Rule Conditions.

Conclusion

I’m very open to suggestions and feedback on this- maybe we can crowdsource a better way, but for now, this seems to be a reasonable alternative to Direct Call Rules. Let me know what you think!

Using Charles Proxy for Analytics

Charles Proxy is a packet-sniffing tool often used by web analysts and developers to debug beacons being sent to Adobe. I find it has some advantages over other tools, even over the Adobe Debugger, because it isn’t looking at your javascript or your “s” object on your DOM, and isn’t tied to a specific browser window or page, but rather it looks at what traffic is actually being sent (and hopefully, successfully received) by Adobe, regardless of where or how it is sent.

charlesFullBeacon

This shows all the parameters (or variables) in your analytics beacon. Some very nice person put up a cheat sheet of how those parameters map to variables or are used in processing and reporting.

On top of looking at Analytics Beacons, I often use it to do the following:

  • Examine which DTM files are loading, and in what order.
  • Check that calls to other Adobe tools (Marketing Cloud ID Service, Target, Audience Manager) are happening in the right order, and with the right parameters.
  • Route traffic from a mobile device through my desktop to see analytics beacons from that device (see instructions below).
  • Use Tools>Map Local to use a file on my machine to replace one of my client’s pages in my browser (for instance, I could save the source of a client’s page, tweak it for my needs, save it to my desktop, and then use Charles Map Local to look at how that page would behave in place of the real one). This makes it easy to see how a client’s page acts when a DTM library is placed on it, or if certain code is removed.
  • Use Tools>Map Remote to replace a file on the client’s site with a file from elsewhere on the internet. Often, this is to swap one DTM library out for another. Sometimes, I do this just to purposefully break a file- if I’m curious how a site would behave without jQuery, I might use map local to say “replace the jQuery file with a file at blank.js” (which doesn’t actually exist, so it effectively just says “don’t load the jQuery file). I did this often enough, and got sick of even seeing 404 errors, I created a blank file specifically for replacing other files.
  • Get the full URL of a POST beacon, so I can use my beacon parser tool to split it out.
  • Save a session of Analytics traffic so you can send it to peers.
  • View the Headers and Cookies attached to a given beacon (at the bottom of a beacon, where you are viewing your params, switch from Query Params to either “Headers” or “Cookies”):
    charlesHeaders

Charles is free if you’re ok with seeing a long splash screen and having sessions limited in length; it’s 50 dollars for a personal license if you want no splash screen and unlimited session length (I bought mine many years and many updates ago, it was worth every penny).

Charles Proxy and I have a bit of a love-hate relationship. It is pretty much ALWAYS open on my machine. It makes many aspects of my job so much easier. But sometimes, it just… doesn’t work. Many folks use the Tools>Rewrite capability to replace just certain code on the page, but it rarely works for me. And getting all the certification set up to accurately track SSL traffic is tricky. So, below is my guide for all the steps I follow when setting up Charles on a new machine to make it ideal for the type of tasks I test to do:

Install it

I prefer the Beta version since it happens to work better with my certificates and such, but for most folks the normal download should do just fine.  After you install it, if you use Firefox, you’ll also want to download the Firefox extension- the easiest way is to go to Charles>Help>Install Mozilla Firefox Add-on. This is not needed for Chrome or the other common modern browsers.

Filter it

Charles captures ALL traffic sent from your machine to anywhere on the internet. It can be… comprehensive. I quickly find it to be too much information, and I use the “Include” tab in Proxy>Recording Settings to tell Charles to only show things relevant to my job. Below are the common ones (asterisks are wildcards):

  • “assets.adobedtm.com” will show all Akamai-hosted DTM files. (If you self-host, you’ll want to add that URL up to the folder that contains your scripts- in other words, stop just before it says “/satelliteLib-“. That way you see ALL DTM files, like your s_code file, not just your main library)
  • “*demdex.net” shows Audience Manager and Marketing Cloud ID calls (fun fact: the Marketing Cloud ID service uses technology gained from Adobe’s acquisition of Demdex).
  • “*.tt.omtrdc.net” shows Target calls
  • Whatever domain you capture your analytics tracking on. For instance, on this site, my non-secure beacons are sent to “jenniferkunz.d1.sc.omtrdc.net”. Make sure you include both the secure and non-secure domains. If in doubt, you may be able to check this by seeing what your s.trackingServer and s.trackingServerSecure variables are set to in your s_code.
  • Because I have many Analytics clients, I also include all the common domains for analytics tracking, but you may not want to do this because you’ll catch many beacons not relevant to your implementation:
    • “*.2o7.net” for non-RDC, non-CNAME implementations
    • “*.omtrdc.net” for RDC implementations
    • “*metric*” as the domain and “*/b/ss*” as the path, to capture a vast majority of CNAME implementations.

charlesInclude

Use “Sequence” Mode

Most folks use “Structure” mode for viewing beacons, where a panel on the left will show all traffic, broken down by domain:

charlesStructureMode

I used this for years before realizing how much more I like sequence mode, which offers a few benefits:

  • It shows things in the order that they occur (making it easy to spot things like “uh-oh, the marketing cloud ID service hadn’t responded by the time my beacon fired”
  • You don’t have to know the root domain of your beacons off the top of your head. Secure and non-secure beacons are shown alongside each other.
  • It makes it very easy to filter. I often filter by “b/ss” to show only Analytics beacons, but you could also filter by “v0” to see only beacons that set the s.campaign variable, or by “landing” to show only beacons that had a pageName including “landing”.
    charlesSequence

Get it working with SSL

By default, Charles can’t read traffic from secure sources- or it can only show them with encrypted values (generally useless for debugging). There is information on the Charles site about setting up SSL tracking, but it can be hard to know what it means for you.

First, you need to set up a Root Certificate on your machine, giving Charles control over how your machine handles SSL Certificates. Go to Help>SSL Proxying>Install Charles Root Certificate. Follow the prompts from your operating system, but keep a careful eye out for two settings:

  • From the Charles site (specific to windows): “The certificate must be imported into the “Trusted Root Certification Authorities” certificate store, so override the automatic certificate store selection.”
  • Ensure the certificate is set to “Always trust” (if you miss this setting first time around, and you’re on a mac, you can change it later by opening your keychain.

If you have Firefox, you’ll also need to install the certificate through Firefox. If you’ve installed the Charles extension, then within Firefox you should be able to go to Tools>Charles Proxy>Install Charles Root Certificate.  When given the option, click “Trust this CA to identify websites”.

All of this has gotten Charles primed to listen for SSL traffic, but you still need to tell Charles which specific sites it can listen to, by going to Proxy>SSL Proxying Settings>SSL Proxying and entering the domains of your SSL analytics traffic. I tend to have this list pretty closely match by list of overall traffic filters, mentioned above (eg: “*2o7.net”, “*omtrdc.net”, “assets.adobedtm.com”, and “smetrics.*.com”).

Use it for Mobile Devices

If you want to route your traffic from a mobile device through Charles, it’s fairly simple. Below are instructions for an iOS device… sorry Android users, you’ll need to google to find similar instructions). Your mobile device and your desktop will need to be on the same network (ie, same WiFi). If your desktop is on VPN, your phone will need to be too.

First, if you will be looking at secure traffic, you’ll need to install the Charles certificate to your device. On your device, go to http://www.charlesproxy.com/getssl and follow the prompt. Pretty easy.

Next, get your local IP address (hint: this is different from your overall IP address, and specific to within your network). Charles makes this easy- in Charles, go to Help>Local IP Address.  You may want to jot this down and close this window, because when you get things working, Charles will want to pop up another window you need to pay attention to.

Lastly, on your phone, open up your WiFi settings and scroll to the bottom. Under HTTP Proxy, select “Manual”, then under Server, enter the Local IP address you noted above. Under Port, enter “8888”. Then click Renew Lease.

charlesMobile

[update- September 2017): If you still have issues with SSL traffic and get a “SSLHandshake: Received fatal alert: unknown_ca” error, you may need to go into your device’s settings- Settings > General > About > Certificate Trust Testings – and make sure the Charles Proxy Custom Root Certificate is marked as trusted.

Don’t forget to switch this back to “off” and re-renew your lease when done. 

Now move out to your browser and do a simple test to see if traffic gets sent and shows in Charles. The first bit of traffic should prompt a Charles warning on your machine, asking you to Allow this traffic.  If you miss this setting the first time around, you can tweak it under Charles>Tools>Access Control Settings.

Troubleshooting

If after all of this, it isn’t working right, here’s a few things to try:

  • If you are on VPN, and Charles isn’t cooperating, try switching the order you do things in: perhaps by opening your VPN first, THEN Charles, or the other way around. Experiment. I
  • If Charles works in one browser and not in another, check your Proxy menu: are you set to capture traffic data both on your OS (and therefore Chrome), as well as Firefox?
  • If Charles is interfering with things like your email server, check out your Bypass Proxy settings.