Glue configuration for optimal logging and cost efficiency

Introduction

In today's cloud-centric computing environment, managing costs while ensuring optimal performance can be quite the balancing act, especially when dealing with large-scale data processing tasks. AWS Glue is frequently employed to prepare and transform data for analytics and streaming use cases. One area where costs can unexpectedly mount is logging, particularly when using services like Amazon CloudWatch. This blog post will guide you through configuring Log4J for Spark on AWS Glue to optimize logging and achieve cost efficiency.

Understanding the Basics

Before diving into the specifics, it’s essential to understand what AWS Glue, Spark, Log4J, and Amazon CloudWatch entail and how they interact:

AWS Glue: A serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and real-time application development.
Apache Spark: An open-source unified analytics engine for large-scale data processing.
Log4J: A reliable, fast, and flexible logging framework (APIs) written in Java, which is used by many different platforms including Apache Spark.
Amazon CloudWatch: A monitoring and observability service built for aggregating logs and metrics from different services.

The Cost of Logging

Logging is crucial for debugging and monitoring applications, but it can lead to high costs if not managed properly. Each log entry consumes storage and, depending on the verbosity of the logs, the cost associated with storing and querying these logs can escalate quickly.

CloudWatch costs can accrue based on several factors:

Log Data Ingestion: CloudWatch charges for each gigabyte of log data ingested. If your applications are highly verbose in their logging, this can result in significant data ingestion charges.
Log Storage: After ingestion, CloudWatch Logs retains this data, charging for the storage per gigabyte per month. The longer you retain logs and the more data you store, the higher the costs.
Data Transfer: While transferring data within the same AWS region is free, transferring log data across regions can incur additional costs.

Configuring Log4J for Spark on AWS Glue

Here’s how you can tailor Log4J on Spark within AWS Glue to trim unnecessary logging and thus reduce costs:

Modify the Log4J Properties File: AWS Glue uses Apache Spark, which in turn uses Log4J for logging. You can customize the logging level by tweaking the log4j.properties file. Lowering the log level from INFO to WARN or ERROR reduces the volume of log entries generated. This directly impacts the cost by decreasing the amount of log data sent to CloudWatch.

  status = error

  rootLogger.level = warn

  # Console Appender
  rootLogger.appenderRef.stdout.ref = STDOUT
  appender.console.type = Console
  appender.console.name = STDOUT
  appender.console.target = SYSTEM_ERR
  appender.console.layout.type = PatternLayout
  appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %p [%t] %c{2} (%F:%M(%L)): %m%n

  # Loggers setting
  # Set the default spark-shell log level to WARN. When running the spark-shell, the
  # log level for this class is used to overwrite the root logger's log level, so that
  # the user can have different defaults for the shell and regular Spark apps.
  logger.spark_repl_main.name = org.apache.spark.repl.Main
  logger.spark_repl_main.additivity = false
  logger.spark_repl_main.level = warn

  logger.spark_deploy_yarn_client.name = org.apache.spark.deploy.yarn.Client
  logger.spark_deploy_yarn_client.additivity = false
  logger.spark_deploy_yarn_client.level = debug

  # Settings to quiet third party logs that are too verbose
  logger.spark_jetty.name = org.spark_project.jetty
  logger.spark_jetty.additivity = false
  logger.spark_jetty.level = warn

  logger.spark_jetty_util_abstract_lifecycle.name = org.spark_project.jetty.util.component.AbstractLifeCycle
  logger.spark_jetty_util_abstract_lifecycle.additivity = false
  logger.spark_jetty_util_abstract_lifecycle.level = error

  logger.spark_repel_expr_typer.name = org.apache.spark.repl.SparkIMain$exprTyper
  logger.spark_repel_expr_typer.additivity = false
  logger.spark_repel_expr_typer.level = warn

  logger.spark_repel_loop_interpreter.name = org.apache.spark.repl.SparkILoop$SparkILoopInterpreter
  logger.spark_repel_loop_interpreter.additivity = false
  logger.spark_repel_loop_interpreter.level = warn

  logger.apache_parquet.name = org.apache.parquet
  logger.apache_parquet.additivity = false
  logger.apache_parquet.level = error

  logger.parquet.name = parquet
  logger.parquet.additivity = false
  logger.parquet.level = error

  logger.sql_datasource_parquet.name = org.apache.spark.sql.execution.datasources.parquet
  logger.sql_datasource_parquet.additivity = false
  logger.sql_datasource_parquet.level = error

  logger.sql_datasource_file_scan_rdd.name = org.apache.spark.sql.execution.datasources.FileScanRDD
  logger.sql_datasource_file_scan_rdd.additivity = false
  logger.sql_datasource_file_scan_rdd.level = error

  logger.hadoop_codec_pool.name = org.apache.hadoop.io.compress.CodecPool
  logger.hadoop_codec_pool.additivity = false
  logger.hadoop_codec_pool.level = error

  # SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
  logger.hive_retrying_hms_handler.name = org.apache.hadoop.hive.metastore.RetryingHMSHandler
  logger.hive_retrying_hms_handler.additivity = false
  logger.hive_retrying_hms_handler.level = fatal

  logger.hive_function_registry.name = org.apache.hadoop.hive.ql.exec.FunctionRegistry
  logger.hive_function_registry.additivity = false
  logger.hive_function_registry.level = error

  # Remove DynamoDB tedious messages
  logger.ddb_page_result_multiplexer.name = org.apache.hadoop.dynamodb.preader.PageResultMultiplexer
  logger.ddb_page_result_multiplexer.additivity = false
  logger.ddb_page_result_multiplexer.level = off

  logger.ddb_read_worker.name = org.apache.hadoop.dynamodb.preader.ReadWorker
  logger.ddb_read_worker.additivity = false
  logger.ddb_read_worker.level = off

  packages = com.amazonaws.services.glue.cloudwatch

  # Progress Bar Appender; Progress bar content will not be added to dockerlog or cx error log
  appender.progress_bar.type = CloudWatchAppenderLog4j2
  appender.progress_bar.name = BAR
  appender.progress_bar.layout.type = PatternLayout
  appender.progress_bar.layout.pattern = %m%n

  appender.progress_bar.filter.bar_level_filter.type = CloudWatchLevelRangeFilter
  appender.progress_bar.filter.bar_level_filter.loggerToMatch = com.amazonaws.services.glue.ui.GlueConsoleProgressBar
  appender.progress_bar.filter.bar_level_filter.minLevel = fatal
  appender.progress_bar.filter.bar_level_filter.maxLevel = warn
  appender.progress_bar.filter.bar_level_filter.onMatch = accept
  appender.progress_bar.filter.bar_level_filter.onMismatch = deny

  loggers = BarLogger
  logger.BarLogger.name = com.amazonaws.services.glue.ui.GlueConsoleProgressBar
  logger.BarLogger.level = error
  logger.BarLogger.additivity = false
  logger.BarLogger.appenderRef.progress_bar.ref = BAR
  appender.progress_bar.flushInterval=5
  appender.progress_bar.maxRetries=5
  appender.progress_bar.logStream=progress-bar
  appender.progress_bar.logGroup=/aws-glue/jobs/logs-v2

Modify Glue job to read the Log4J configuration

Now you need to change the Glue job to inject the log properties into Spark configuration.
- If you are using AWS console, you need to write the properties file path in the Referenced files path part:

If you are using CDK, you can pass as the --extra-files parameter:

  glue_alpha.Job(
              self,
              "GlueJob",
              default_arguments={
              "--extra-files": f"s3://{artifacts_bucket.bucket_name}/log4j2.properties"
              }
          )

Conclusion

Optimizing your logging strategy within AWS Glue by customizing Log4J for Spark applications is a practical step towards managing cloud expenditure effectively. By fine-tuning the logging levels and adopting a more strategic logging approach, you can significantly reduce costs while maintaining the necessary visibility into your applications’ performance and health. Regular monitoring and adjustments ensure that your logging remains both effective and economical.

If you are interested to get more insight about your cloud costs, you can check how DataChef implemented its own CDK Budget monitoring constructs.