Empty /wafr-accelerator On EC2: How To Fix?

by Alex Johnson 44 views

Experiencing an empty /wafr-accelerator directory on your EC2 instance within the AWS Well-Architected Acceleration project can be frustrating. This article provides a comprehensive guide to troubleshoot and resolve this issue, ensuring your environment is correctly set up and functioning as expected. We will explore the common causes behind this problem and provide step-by-step instructions to verify and, if necessary, manually trigger the necessary processes to populate the directory.

Understanding the Issue

When working with the AWS Well-Architected Acceleration with Generative AI project, the /wafr-accelerator directory on your EC2 instance should contain the UI application scripts. As noted in the project's documentation and related discussions, these scripts are typically downloaded onto the instance using an SSM (Systems Manager) run command initiated by a Lambda function post-build. This process might take a short time to complete.

However, if you find the directory empty even after an extended period (e.g., 3 hours), as the user reported, it indicates that the SSM run command might not have executed successfully or that there might have been an issue during the script download process. Let's dive deeper into potential causes and solutions.

Diagnosing the Problem

Before attempting any manual intervention, it's crucial to diagnose why the scripts are missing in the first place. Here are several factors to consider:

  1. SSM Agent Status: The SSM Agent must be running on your EC2 instance and be able to communicate with the AWS Systems Manager service. If the agent is not running or is unable to connect, the run command will fail.
  2. IAM Permissions: The EC2 instance role must have the necessary IAM permissions to allow SSM to execute commands on the instance. This includes permissions to access S3 buckets (if the scripts are downloaded from S3) and write to the file system.
  3. Lambda Function Execution: The Lambda function responsible for triggering the SSM run command might have failed or encountered an error. Check the Lambda function's logs in CloudWatch for any error messages.
  4. Network Connectivity: The EC2 instance needs outbound internet access (or a route to the appropriate VPC endpoint) to download the scripts from S3 or any other external source.
  5. Timing Issues: Although unlikely after 3 hours, there could have been temporary network glitches or delays in the AWS infrastructure that prevented the successful execution of the SSM run command.

Step-by-Step Troubleshooting

Follow these steps to identify and resolve the issue:

1. Check SSM Agent Status

Connect to your EC2 instance via SSH and run the following command to check the status of the SSM Agent:

sudo systemctl status amazon-ssm-agent

If the agent is not running, start it using:

sudo systemctl start amazon-ssm-agent

Also, ensure that the agent is enabled to start on boot:

sudo systemctl enable amazon-ssm-agent

2. Verify IAM Permissions

Go to the IAM console and check the IAM role associated with your EC2 instance. Ensure that the role has the following policies attached (or equivalent custom policies):

  • AmazonSSMManagedInstanceCore: This policy provides the basic permissions required for SSM Agent to function.
  • AmazonS3ReadOnlyAccess: If the UI application scripts are stored in an S3 bucket, this policy (or a more restrictive policy granting read access to the specific bucket) is required.
  • A custom policy allowing SSM to write to the /wafr-accelerator directory. For example:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ssm:UpdateInstanceInformation",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket-name",
                "arn:aws:s3:::your-bucket-name/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:SendCommand",
                "ssm:GetCommandInvocation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogStreams"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }
    ]
}

Replace your-bucket-name with the actual name of your S3 bucket.

3. Check Lambda Function Logs

Identify the Lambda function responsible for triggering the SSM run command. Go to the CloudWatch console and view the logs for that function. Look for any error messages or exceptions that might indicate why the command failed to execute. Common errors include:

  • IAM permission issues (Lambda function not authorized to execute SSM commands).
  • Invalid SSM document name.
  • Incorrect parameter values.
  • Network connectivity problems.

4. Verify Network Connectivity

Ensure that your EC2 instance has outbound internet access or a route to the appropriate VPC endpoint for S3 and SSM. You can test internet access by running:

ping www.google.com

If you are using VPC endpoints, verify that the endpoints are correctly configured and that your EC2 instance is using the correct route table.

5. Manually Trigger the SSM Run Command

If the previous steps haven't revealed any obvious issues, you can try to manually trigger the SSM run command. This will help determine if the problem lies with the Lambda function or with the SSM execution itself. Here's how you can do it:

  1. Identify the SSM Document: Determine the name of the SSM document used to download the UI application scripts. This information should be available in the Lambda function's code or configuration.
  2. Use the AWS CLI: Use the AWS CLI to execute the aws ssm send-command command. You'll need the instance ID of your EC2 instance and the name of the SSM document. You might also need to provide parameter values, depending on the document's requirements.

Here's an example command:

aws ssm send-command \
    --instance-ids "i-xxxxxxxxxxxxxxxxx" \
    --document-name "YourSSMDocumentName" \
    --parameters "{\"SourceS3Bucket\":[\"your-bucket-name\"], \"SourceS3Key\":[\"path/to/your/script.zip\"], \"DestinationPath\":[\"/wafr-accelerator\"]}" \
    --output text \
    --query "Command.CommandId"

Replace the following placeholders with your actual values:

*   `i-xxxxxxxxxxxxxxxxx`: Your EC2 instance ID.
*   `YourSSMDocumentName`: The name of the SSM document.
*   `your-bucket-name`: The name of the S3 bucket containing the scripts.
*   `path/to/your/script.zip`: The path to the script package within the S3 bucket.

Note: The parameters you need to provide will depend on the specific SSM document being used. Check the document's definition to determine the required parameters.

  1. Check Command Status: After executing the command, the CLI will return a command ID. You can use this ID to check the status of the command:
aws ssm get-command-invocation \
    --command-id "your-command-id" \
    --instance-id "i-xxxxxxxxxxxxxxxxx"

Replace your-command-id with the command ID you received in the previous step.

The output will show the status of the command (e.g., Pending, InProgress, Success, Failed) and any error messages.

Alternative Solutions and Considerations

  • CloudFormation: If your infrastructure is managed by CloudFormation, review the CloudFormation stack events for any errors related to the EC2 instance creation or the execution of custom resources.
  • AWS Support: If you've exhausted all troubleshooting steps and are still unable to resolve the issue, consider opening a support case with AWS Support. They can provide deeper insights into your environment and assist with more advanced troubleshooting.

Conclusion

An empty /wafr-accelerator directory on your EC2 instance can stem from various issues, ranging from SSM Agent problems to IAM permission errors and network connectivity glitches. By systematically following the troubleshooting steps outlined in this article, you should be able to pinpoint the root cause and implement the necessary corrective actions. Remember to verify each component of the process, from the SSM Agent to the Lambda function and the EC2 instance's configuration, to ensure a smooth and successful deployment of the UI application scripts.

By understanding these potential issues and proactively addressing them, you can ensure your AWS Well-Architected Acceleration project runs smoothly and efficiently. If you are still facing issues, then consult with AWS's SSM Agent Documentation.