Streamlit in Production: How to Stream Logs to AWS CloudWatch

Streamlit: Stream Logs to AWS CloudWatch
Streamlit: Stream Logs to AWS CloudWatch

If you’re running a streamlit application in production, you might want to ship logs generated by it to cloudwatch. If you’re using ECS then it might be simple but if your have deployed your app on an EC2 instance or Kubernetes then it might not be that straightforward.

Effective logging is key for monitoring and debugging. In this guide, we'll go step by step to explain how to set up cloudwatch logger in a Streamlit application. By the end, you'll have a fully functional setup that logs application events, errors, and user inputs directly to CloudWatch for monitoring and analysis.

Prerequisites

  1. AWS Account: You'll need an AWS account to set up a CloudWatch log group.
  2. Python Environment: Set up a Python environment with the required packages.
  3. AWS CLI Configured: Ensure you have the AWS CLI installed and configured on your machine.

Step 1: Installing the Required Libraries

Here are the libraries and modules we’ll be using in this project:

  1. Streamlit: This is the framework used to build interactive web applications in Python. It allows for quick creation of web apps with simple syntax.
  2. boto3: This is the official AWS SDK for Python. It allows us to interact with AWS services like CloudWatch. In this case, we use it to programmatically set the log retention policy for CloudWatch logs.
  3. watchtower: This library makes it easy to ship logs to AWS CloudWatch from Python applications. We use it to create a CloudWatch log handler that directs our logs to a specific CloudWatch log group.
  4. logging: Python’s built-in logging module provides a flexible framework for logging. We use it to define our logging behavior, such as sending log messages to CloudWatch through the watchtower handler.
  5. sys: We override the built-in print function using Python's sys.modules. By doing this, we ensure that all print statements also generate logs and send them to CloudWatch in addition to printing them in the terminal or console.

Here’s the requirements.txt file containing the versions of each package:

streamlit==1.39.0
boto3==1.35.40
watchtower==3.3.1

Install these packages using:

pip install -r requirements.txt

Step 2: Creating a CloudWatch Log Group

Before we begin coding, we need to create a log group in AWS CloudWatch where logs will be stored. Follow these steps:

  1. Go to CloudWatch in the AWS Management Console.
  2. Navigate to Logs and click Log groups.
  3. Click Create log group, and named it as per you requirement here my log group name is Option1_streamlit_log.

This creates a log group where all your logs will be stored.


Step 3: The Full Code with Explanation

Below is the code that connects all these pieces:

import streamlit as st
import logging
import sys
import boto3
from watchtower import CloudWatchLogHandler

# Configure logging
log_group_name = 'Option1_streamlit_log'  # Replace with your log group name
log_stream_name = 'streamlit-logs'  # You can customize this name
retention_days = 5  # Retention policy: 5 days

# Set up CloudWatch logging
cloudwatch_handler = CloudWatchLogHandler(log_group=log_group_name, stream_name=log_stream_name)
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logger.addHandler(cloudwatch_handler)

# Custom print function to log to CloudWatch
def print_to_cloudwatch(*args, **kwargs):
    message = " ".join(str(arg) for arg in args)
    logger.info(message)  # Log the message to CloudWatch
    sys.__stdout__.write(message + "\n")  # Output to console

# Replace the built-in print function
sys.modules['builtins'].print = print_to_cloudwatch

# Set log retention policy to 5 days
def set_log_retention():
    client = boto3.client('logs')
    client.put_retention_policy(
        logGroupName=log_group_name,
        retentionInDays=retention_days
    )
    print(f"Log retention policy set to {retention_days} days for log group {log_group_name}")

# Example Streamlit application
def main():
    st.title("My Streamlit App")
    
    # Example usage of print
    print("Streamlit app started.")
    
    # Set log retention when the app starts
    set_log_retention()
    
    # Text input box
    user_input = st.text_input("Enter something:")
    
    if user_input:
        print(f"User input received: {user_input}")
        st.write(f"You entered: {user_input}")
    else:
        print("Waiting for user input...")

if __name__ == "__main__":
    main()

How the Code Works

  1. Logging to CloudWatch: The CloudWatchLogHandler is created with the CloudWatch log group and stream name. It’s connected to the logger so that every log created is sent to CloudWatch.
  2. Custom Print Function: We’ve overridden the built-in print function with print_to_cloudwatch, which ensures that every time print is called, the message is sent to CloudWatch and also printed to the console.
  3. Log Retention Policy: The function set_log_retention() ensures that logs are retained for only 5 days. This is done using the boto3 client for CloudWatch Logs, which sets the retention policy.
  4. Streamlit App: This is a basic interactive Streamlit app. It takes user input from a text box and logs each input or action using the print statements, which are automatically logged to CloudWatch.

Step 4: Running the Application

To run the Streamlit app, use the following command in your terminal:

streamlit run app.py

The app will start on http://localhost:8501. When you interact with the app, you’ll see the corresponding logs in AWS CloudWatch.

Step 5: Verifying Logs in CloudWatch

  1. Go to the AWS CloudWatch Console.
  2. Navigate to Log groups and select your log group Option1_streamlit_log.
  3. Open the Log streams, where you’ll find streamlit-logs containing the logs from your app.

Conclusion

By using libraries like watchtower and boto3, it becomes easy to integrate logging with AWS CloudWatch. This allows for efficient logging of application events, user inputs, and errors. With a few lines of code, you can ensure that every print statement in your Streamlit app is logged to AWS CloudWatch for future reference and debugging.

Are you looking to optimise workflows for your Streamlit Applications? Reach out to us at KubeNine Consulting, we can help!