21 Questions to Ask Before Hiring a Conversion Rate Optimization (CRO) Agency

Reading Time: 12 minutes

Hiring a Conversion Rate Optimization (CRO) agency is a big decision. Investing in AB testing and Conversion Rate Optimization does not guarantee results. In fact, working with the wrong organisation can put a big dent in your marketing budget, without increasing your revenue…

This list of 21 questions will help you to avoid that situation, telling you everything you need to know about an agency before you hand over a big slice of your marketing budget. 

There are now well over 100 conversion rate optimization agencies in the UK. Choosing the right one can save you money in the short-term and make you much more in the long-term.

Choosing a CRO agency

• Start by making a list of agencies and contact at least 5 of them, asking to speak to a sales representative.
• A good sales person will want to understand what you are looking for first. Tell them that you want to “increase your sales” and let the sales representative present their company’s services to you.
• Once you have begun the conversation, ask the first question …

Question 1: Is CRO your speciality?

Organisations that try to do everything rarely provide an effective service. You need a specialist conversion rate optimization agency. That does not mean that a consultant can’t offer additional services (such as web design, copywriting or front-end development.) However, these should be used to make their Conversion Rate Optimization and AB testing services more effective.

Question 2: What do you need to know about my company?

The person you are talking to should try to find out as much as possible about your company. If this is not the case, alarm bells should already be ringing. If you only receive re-assurance about the agency’s ability to do “whatever you want,” it may be time to move to the next company on your list.

Question 3: Are you data-driven?

Many marketing activities are based on experience and instinct… Conversion Rate Optimization is NOT one of them. Conversion Rate Optimization agencies base their work on hypotheses, marketing theory and proven techniques. Even so, a trustworthy agency should perform rigorous tests and complex analysis to justify their activities.

If you meet an “expert” who dismisses the importance of data and ignores your KPIs, excuse yourself politely and find someone else. 

Question 4: What is the most important factor in shaping consumer choices?

Consumer choice is not just about how much things cost. Therefore, conversion rate optimization requires more than simply pricing your products strategically.

You need a team of experts who are curious about consumer psychology. Mention Robert Cialdini’s The Psychology of Persuasion, Daniel Kahneman’s Thinking Fast & Slow, or Dan Ariely’s Predictably Irrational, and see what kind of reaction you get.

Question 5: What is the least successful project you have worked on?

Any conversion rate optimization agency can provide examples of projects with outstanding results (+ 20% or 30% uplift). But these examples do not guarantee the success of your project, because your circumstances are unique.

An agency that has been practicing conversion optimization for a long time should be able to talk about difficult projects. In fact, any form of consulting business, even the largest firms such as McKinsey, encounter failure.  If an agency claims never to have experienced setbacks, it is either very new to the industry or it is lying.

Evaluate your consultant’s responses to the question by analyzing:

• The reasons they give for the project’s failure.
• How they managed the project and their client.

The best agencies are obviously those who learn from their setbacks and reflect on them.

Question 6: What optimization tactics do you use?

Effective agencies understand the difference between tactics and strategy. Tactics are simply the tools available to a practitioner. Strategy, is the process of analysing, prioritising and planning. In order to conduct a successful conversion rate optimization project, your agency must think strategically.

To put it another way, there are countless tactics for optimizing a website. Without strategy, an agency might simply try each of them in turn to see if anything works. This would be an inefficient and disruptive approach, without a reliable way to track and achieve permanent results.

If the team you are talking to gives you their “top tips” and “best practices,” but do not describe a structured process, you are talking to the wrong people.

Question 7: How many AB tests should I run each month?

AB testing is only effective if it is based on a strong hypothesis. However, businesses are sometimes encouraged to conduct extensive and unnecessary testing.

In order to standardise their offers and increase their profitability, agencies offer “plans” with a fixed number of AB tests. These programs involve numerous tests over an extended period (during which you will be paying the agency’s fees.)

Similarly, AB testing providers (AB Tasty, VWO, Optimizely) also offer their services in the form of “plans.” It suits them because the more tests you perform, the longer you will be using their software.

On its own, AB testing will not produce results. The important thing is to experiment with treatments that are likely to produce results.

Question 8: Which AB testing tools do you use? 

A Conversion Rate Optimization agency must use a range of tools. AB Tasty, Optimizely or Convertize are good AB Testing tools, but there are countless ab testing tools available.

For example, if you are working on Google Optimize 360, your agency must also be able to perform AB tests on Google Optimize. It is important that they are not “trapped” by association with a competing solution.

Beyond AB Testing software, the best conversion rate experts use a range of other CRO tools:

• Heatmaps: Hotjar, Contentsquare…
• Customer Journey and Funnel Analysis: Amplitude, Mixpanel or Heap Analytics…
• Feedback Systems and questionnaires: Qualaroo…

They should be able to quote them and explain to you in which contexts they will use them.

Question 9: How do you choose which page to optimize first?

Prioritisation is the only way to complete a project efficiently. Keep in mind that optimizing any page is an intensive process requiring data analysis, consumer research, and AB testing. Choosing which page to optimize first, and creating a “Conversion Roadmap,” will help keep your project focused. 

This can be translated into 2-4 weeks of work for each page. If the conversion rate optimization agency chooses the wrong page to optimize first, it will cost you time and money.

To answer Question 9, the agency must explain:

• How they will create a Conversion Roadmap
• How they will decide which pages to test, and in what order

Question 10: How do you identify the best changes to make on a page?

Optimization requires an understanding of both consumer behaviour and web design. Here, too, you are looking for your agency to display a methodical approach.

You need to know:

• How they will decide which elements should be modified and tested
• How this process will be implemented on your site

AB Testing and Optimization agencies assure you that they are experts following a “rigorous process.” But if you are trusting someone with your website, you need as much information as possible. This includes knowing how they will interpret your site and evaluate each page. 

Question 11: What qualitative research do you perform during a project?

Hypotheses should be guided by qualitative research. It is the only way to explain “why” a website does or does not convert. The explanations provided through qualitative research can then be tested and applied.

A serious agency will expect to perform qualitative research in advance of running AB tests. This is so that they can gain a fuller understanding of your visitors. The methods involved in qualitative research include:

• Online questionnaires
• User tests
• Customer interviews
• Exit surveys
• Focus groups (although their value is debated)

Avoid AB Testing and Conversion Optimization agencies that ignore qualitative research, no matter what excuses they give you.

Question 12: What quantitative research do you plan on doing?

Analytics tools provide crucial data about your site, as well as vast amounts of unusable information. Collecting and interpreting these numbers allows you to establish discrete facts: What do visitors do? Which pages convert best? Where does the site lose most visitors?

An optimization agency must be able to gather this information and turn it into meaningful insights. It must also detect any important trends that affect your site’s performance.

You should therefore evaluate the Agency’s ability to perform advanced analyses, beyond the basic clicks and bounces. This includes:

• Data segmentation
• Funnel and goals analysis
• Heatmap analysis

Question 13: Can you test multiple scenarios at once?

The best Conversion Rate Optimization agencies find ways to conduct reliable tests in the least amount of time. One way to do this is by testing multiple scenarios simultaneously. Not only does this allow for faster testing, but it also tests combinations between variables rather than isolated changes.

Your agency should be able to explain whether or not it is possible to save time through multi-variate testing. To do so requires sufficient traffic and advanced analytics systems.

Question 14: How do you optimize for mobile browsing?

More than half of all internet traffic comes from mobile devices. Data from January 2018 suggests there may be as many as 3.8 billion unique users of mobile internet.

An effective agency will account for the different requirements of mobile and desktop internet usage. Using conventional Analytics platforms, you can identify:

• What percentage of visitors come to your site from a mobile device
• The conversion rate of visitors on Mobile compared to visitors on Desktop

Mobile site optimization requires different tactics to desktop optimization. An agency that does not explain how it distinguishes between the two is certainly not experienced enough. 

Question 15: Who will manage the implementation of AB testing on my site?

Some variables require design or development skills. To perform a split test, you will have to create a new version “B” of the page you want to test. But who will create this new page?

The answer depends on the complexity of the new variable.

If the changes are simple, such as a change of title, colour or image, version “B” of a page can be created by anyone with access to the AB Testing tool. 

More complex changes, such as adding filters to a search page or modifying the structure of your site, require access to your site’s code. This means that your development team (Design and IT), the CRO agency, or an external web developer, will have to create it.

Question 16: What do you need from my team?

Conversion rate optimization can be a complicated process. As previously explained, optimization services may require the intervention of your design team and web developers.

You must clearly evaluate how much of your company’s resources will be required to complete the CRO project. Here are some questions to ask:

• In what form will the new designs be delivered to our team; do you use HMTL, PSD or simple Wireframes?
• Do you have Front-End resources to help our team launch AB tests, or will we have to assign them internally?

Question 17: Are there any additional costs beyond your own consulting fee?

CRO can involve hidden costs. To guarantee the success of your project, you need to know what all the associated costs will be in advance.

Here are some of the common additional costs associated with CRO projects:

• The conversion rate optimization agency will probably ask you to install an AB Testing tool on your site. Some AB Testing software can be expensive, and rates often depend on the volume of visitors. Solutions like AB Tasty or Optimizely quickly run over £1000 per month and may require an annual commitment.
• Additional tools such as Heatmaps, Survey platforms and Funnel Analysis platforms, can also come at a significant cost. 
• Any investment of time and effort from your development or design teams counts as a staff cost. Depending on how complicated the testing program is, this could be a significant part of the eventual expense. Any interference with the user experience of your site can cost you conversions. If an agency uses inefficient testing software, you could even find your website performance impaired.

Ask for a complete estimate from your consultant before signing up to a program.

Question 18: What experience do your CRO consultants have?

Conversion optimization requires expertise, versatility and patience. A Conversion Manager (or CRO Manager) must obviously be an expert in Analytics and Consumer Psychology, but they must also have some knowledge of HTML, web design, SEO and digital marketing. The list of necessary skills is long.

Even if the person you are talking to is an expert in CRO, you should also ask for the CV of the people who will manage your project. The success of your conversion optimization project will depend primarily on the people who are actively engaged on it. 

The best Conversion Rate Optimization agencies will only assign qualified agents to your project. If you are not sure this is the case, look for another agency.

Question 19: What are the 2 most important issues on my homepage?

A CRO expert should be systematic and goal-orientated. Ideally, this question should be put to the CRO agent who will be working on your site. Asking this practical question is an important step to take before engaging an agency. Seeing how an agent examines a live page will give you a good indication of the way they will address your Conversion Rate goals.

The purpose of this question is not to demonstrate your agent’s knowledge, but rather to reveal the way they think about a client’s site. There are three important things to consider:

• Are they asking relevant questions to get more information before answering? For example, for a landing page, “how much traffic does this page receive?”
• Are they thinking systematically or focusing on trivial elements (such as the colour of a particular button or a specific image)
• Are they considering your Value Proposition, which is the most relevant and important content on any homepage?

If your agent does not ask relevant questions, does not approach the page in a systematic way, or fails to see your website in the context of your business, the agency is a clear NO-GO. 

Question 20: Can you guarantee an increase in my conversion rate?

Nothing can be taken for granted with Conversion Rate Optimization. The purpose of this question is simple. If a conversion rate expert or optimization agency guarantees you results, run in the opposite direction.

Many factors influence the results of an optimization program, some of which are not under the control of the agency.

For example, your conversion rate will depend on the quality of the traffic that arrives on your site. If your Marketing team decides to launch a Native Advertising campaign, while your previous visitors came from Search Adwords campaigns, the search intentions of your traffic will be affected. This will dramatically alter your conversion rate.

It would be naïve (or dishonest) for an agency to guarantee an increase in your conversion rate.

Question 21: How will you measure the success of this project?

A CRO project should have clearly defined goals. If you hire a conversion optimization agency, you will want to see evidence of an increase in the conversion rate of your website. A good agency should establish and monitor performance indicators (KPIs) that reflect your goals.

• For an e-commerce site, the ultimate goal should be to increase sales without reducing your margins. This means that any AB testing should be configured accordingly. The important effects to observe are those relating to sales. If a CRO agent attempts to convince you that micro-conversions (such as the number of clicks on a shopping cart button) are sufficient, choose someone else.
• For a SaaS site, there are probably two goals that you need to pursue: 1. The number of visitors who convert to subscribers (for example to the free version of your software) 2. The number of subscribers who convert to become a paid user.
• For a lead generation site, it’s … when the lead is generated.

These measurements assume that your site has enough traffic to provide statistically significant results. For sites with too little traffic to achieve this, the measurement of success will rely on interpreting qualitative data. 

However, measuring success is not just about immediate goals. There are other ways to think about the success or failure of a project. A CRO programme can also be evaluated in terms of:

• The transfer of skills to your team.
• Completion of a structured process defined at the start of the project.

Consultants, Conversion Rate Optimization or A/B Testing: What’s Best for your Business?

CRO is an increasingly significant practice in the world of eCommerce. It provides a direct, effective and affordable alternative to traditional consulting services. This means that even small businesses can increase their conversion rate with the help of agency expertise.

It is also possible to conduct optimization on an eCommerce website without hiring a Conversion Rate Optimization agency. A/B Testing software, when combined with a user-friendly knowledge base, provides the means to systematically optimize any website. Explore our archive of expert content for guides on how to perform in-house optimization, including how to optimize a landing page.

For in-depth guides and advice on Conversion Rate Optimization, explore our library of articles from CRO specialists.