Enhancing Adobe Marketo Engage Data Analysis with AWS Glue Integration

Today’s digital-first, B2B landscape presents marketers with complex challenges as they navigate sophisticated buyer journeys involving diverse decision-making groups. Adobe Marketo Engage offers a comprehensive marketing hub for orchestrating cross-channel campaigns. Using AI-driven personalization, automation, and real-time analytics, it helps businesses acquire and retain customers throughout their buying journeys. Marketo Engage empowers B2B marketers to navigate modern complexities and successfully drive measurable business growth through multi-channel engagement, automated customer journeys, and sales-marketing collaboration.

To further enhance their B2B marketing capabilities, organizations are now looking to fully use their marketing data for more informed decision-making and strategy optimization. Recognizing the need to simplify the analytics pipeline, AWS introduced software as a service (SaaS) connectivity for Marketo Engage through AWS Glue, delivering insights faster to enable data-driven decisions. The agile, serverless nature of AWS Glue meets a range of data analytics needs while reducing costs. This powerful integration links the robust marketing automation features of Marketo Engage with AWS’s advanced analytics ecosystem. By seamlessly connecting the platforms, businesses can extract greater value from marketing data, gaining deeper insights into customer behavior and campaign performance. Together, AWS Glue and Marketo Engage unlock new possibilities for data-driven marketing:

• Marketing-sales alignment – Helps automate the transfer of lead and opportunity data between Marketo Engage and CRM systems, making sure that sales and marketing teams are aligned and responsive to customer needs
• Enhanced analytics – Connects Marketo Engage with business intelligence (BI) tools for data-driven campaign optimization, allowing marketers to extract meaningful insights and make informed decisions
• Data integrity – Maintains consistent, high-quality data across all systems, providing reliability and accuracy in marketing and sales operations
• Improved lead quality – Refines lead scoring processes by using the advanced analytics capabilities of AWS, resulting in better-qualified leads and improved sales conversions
• Unified customer view – Provides comprehensive customer insights using enriched AWS datasets for Marketo Engage, offering a holistic understanding of customer interactions and behaviors

In this post, we show you how to use AWS Glue to extract data from Marketo Engage for data processing and enrichment on AWS for use in marketing analytics workflows.

Solution Overview

We explore a use case in which a company wants to run analysis for campaign leads in multiple countries. The resulting leads will be shared with the respective regional marketing representatives. The solution uses AWS Glue to extract data from Marketo Engage and save it in an Amazon Simple Storage Service (Amazon S3) bucket. The following diagram illustrates the solution architecture.

In the following sections, we walk through the high-level steps to implement the solution:

1. Create AWS resources to connect to Marketo Engine and store data.
2. Create an AWS Glue connection.
3. Create an extract, transform, and load (ETL) job using AWS Glue Studio.
4. Analyze the data.

Prerequisites

To set up the integration between Marketo Engage and AWS, the following components are required:

• A Marketo Engage account – If you don’t already have one, create a Marketo Engage application and record the Munchkin ID, client ID, and client secret for the application. Refer to the Marketo Engage developer portal to set up the connection.
• An AWS Glue database – This will serve as the data interaction interface on AWS. The database will expose the data residing in Amazon S3 as queryable AWS Glue tables. For this post, our database is called marketodb.

Create AWS resources to connect to Marketo Engage and store data

We use an AWS CloudFormation template to create an S3 bucket to store data, an AWS Secrets Manager secret for Marketo Engage that the AWS Glue connection needs, and an AWS Identity and Access Management (IAM) role to access the secret. Complete the following steps:

1. Click Launch Stack below.
   
2. On the Specify stack details page, enter a name for the stack.
3. Specify the Marketo client secret.
4. Choose Next.
5. On the Configure stack options page, choose Next.
6. On the Review page, select I acknowledge that AWS CloudFormation might create IAM resources.
7. Choose Submit. Note: The stack takes about 2 minutes to deploy.
8. After the stack is created, make a note of the S3AccessRoleARN You will need this to create the Marketo Engage connection.

Create an AWS Glue connection

Complete the following steps to create an AWS Glue connection:

1. On the AWS Glue console, choose Data connections in the navigation pane.
2. Choose Create connection.
3. For Data sources, select Marketo.

4. Enter the Adobe MUNCHKIN_ID.
5. Choose the IAM role created in the previous section as the AWS Glue connection IAM service role.
6. Provide the Adobe ClientId as the user-managed client application client ID.
7. Provide the Secrets Manager secret you created earlier.
8. Choose Next.
9. Specify your preferred connection name.
10. Choose Next.
11. Review the settings, then choose Create connection.

Create an ETL job using AWS Glue Studio

Complete the following steps to create an ETL job:

1. On the AWS Glue console, choose ETL jobs in the navigation pane.
2. Choose Create job.
3. Choose Visual ETL.
4. Add Marketo as a source node.
5. Add Amazon S3 as the target node.
   

6. Choose the Marketo Engage data source node, and the editor will show a configuration pane on the right side of the diagram.
7. In the Data source properties pane, provide the following information:
   1. For Name, enter a name (for example, Marketo).
   2. For Marketo connection, choose your Marketo Engage connection.
   3. For Entity Name, choose Leads as the entity to retrieve from Marketo Engage.
   4. For Fields, choose All Fields as the fields to retrieve from Marketo Engage.
   5. For Filter, enter gender=’Male’ to pull leads according to the campaign criteria. Note that in this blog post you’re using a synchronous mode in which the Marketo Adobe API limits require that the retrieved data set is less that 1000. See the AWS documentation to apply the criteria and mechanisms that support your campaigns.

You can observe the data preview pane reflecting the modifications you have made.

8. Choose the Amazon S3 target node to configure it.
9. In the Data target properties pane, provide the following information:
   1. For Name, enter a name (for example, Amazon S3).
   2. For Node parents¸ choose Marketo.
   3. For Format, choose Parquet.
   4. For Compression Type¸ choose Snappy.
   5. For S3 Target Location, enter the path to the S3 bucket you created earlier, and optionally specify a prefix. This will inform the ETL job where to store the data retrieved from Marketo Engage.
   6. For Data Catalog update options, select Create a table in the Data Catalog and on subsequent runs, update the schema and add new partitions.
   7. For Database, choose your database in the AWS Glue Data Catalog.
   8. For Table name, enter a table name for the Data Catalog (for example, marketo_leads).

After you configure the source and target nodes, both nodes in the Visual ETL Editor should have a green check mark, indicating they are correctly configured.

10. Specify the name for the job and save it.
11. When the job is saved, choose Run to invoke the ETL job.
12. After the job starts, go to the Runs tab and observe the run until completion.

Depending on the size of the data in your account object in Marketo Engage, the job will take a few minutes to complete. After a successful job run, a new table called marketo_leads is created and populated with data from Marketo Engage.

Analyze the data

After a successful run, you can now use Amazon Athena analyze the data from Marketo Engage with the data residing on AWS. If you’re using Athena for the first time, refer to Create a query output location for instructions to set up the query editor. Then run the following query:

SELECT country, COUNT(*) as count FROM marketo_leads GROUP BY country ORDER BY country;

The query will output the number of people within each country who can be contacted as targeting leads for campaigns, and you can enrich this output by adding other datasets in your data lake or data warehouse. You can expect to see an output like the following screenshot.

Clean up

To avoid incurring charges, clean up the resources in your AWS account by completing the following steps:

1. Delete the table created from the Data Catalog:
   1. On the AWS Glue console, navigate to the Data Catalog.
   2. Select the table and choose Delete.
2. Delete the ETL job:
   1. On the AWS Glue console, choose ETL jobs in the navigation pane.
   2. From the list of jobs, select the job you created, and on the Actions menu, choose Delete.
3. Delete the data connection:
   1. On the AWS Glue console, choose Data connections in the navigation pane.
   2. Select the Marketo Engage connection from the list of connectors, and on the Actions menu, choose Delete.
4. Delete the CloudFormation stack:
   1. On the CloudFormation console, choose Stacks in the navigation pane.
   2. Select the stack you created for the S3 bucket and related resources and delete it.

Conclusion

The AWS Glue connector for Marketo Engage streamlines data integration, permitting seamless data synchronization between Marketo Engage and AWS services for a holistic view of customer information. This powerful integration enhances the capacity for advanced analytics, enabling marketers to glean precise and insightful learnings from their data; these insights can then be used to inform and refine marketing strategies, boosting campaign performance and driving better business outcomes

For more information on the AWS Glue connector for Marketo Engage and AWS Glue, refer to the relevant user guides and visit the AWS Glue website.

About the Authors

Kenny Rajan is a Principal Enterprise Architect at AWS specializing in integrating generative AI with enterprise systems like SAP and Adobe. He helps organizations modernize their digital experience platforms and supply chain and back-end systems through data and AI-powered cloud solutions. Outside of work, he contributes to technology education and charitable initiatives.

Rafał Pawłaszek is a Senior Cloud Application Architect at AWS. Rafał supports customer transformation to the cloud and customer enablement in the cloud. Outside of work, he is interested in astronomy, astrophysics, and psychology, and loves spending time with family.

Basheer Sheriff is a Senior Solutions Architect at AWS. He loves to help customers solve interesting problems using new technology. He is based in Melbourne, Australia, and likes to play sports such as football and cricket.

Kamen Sharlandjiev is a Sr. Big Data Solutions Architect, Amazon MWAA and AWS Glue ETL expert. He’s on a mission to make life easier for customers who are facing complex data integration and orchestration challenges. His secret weapon? Fully managed AWS services that can get the job done with minimal effort. Follow Kamen on LinkedIn to keep up to date with the latest Amazon MWAA and AWS Glue features and news!