Amazon EMR Containers

接入教程

1. 创建Amazon EKS集群
2. 配置IAM角色和权限
3. 部署EMR on EKS作业
4. 提交Spark或Hive作业
5. 监控作业运行状态
6. 清理作业和资源

使用Python创建EMR容器虚拟集群

python

import boto3
from botocore.config import Config

# 配置客户端
config = Config(region_name='us-east-1')
emr_client = boto3.client('emr-containers', config=config)

# 创建虚拟集群
response = emr_client.create_virtual_cluster(
    name='my-analytics-cluster',
    containerProvider={
        'type': 'EKS',
        'id': 'your-eks-cluster-id'
    }
)

print(f"Virtual cluster created: {response['id']}")

使用PHP提交EMR容器作业

php

<?php
require 'vendor/autoload.php';

use Aws\EmrContainers\EmrContainersClient;
use Aws\Exception\AwsException;

$client = new EmrContainersClient([
    'region' => 'us-west-2',
    'version' => 'latest'
]);

try {
    $result = $client->createJobRun([
        'virtualClusterId' => 'your-virtual-cluster-id',
        'name' => 'spark-job-example',
        'executionRoleArn' => 'arn:aws:iam::account-id:role/EMRContainers-JobExecutionRole',
        'releaseLabel' => 'emr-6.2.0-latest',
        'jobDriver' => [
            'sparkSubmitJobDriver' => [
                'entryPoint' => 's3://bucket/code/spark-example.py',
                'sparkSubmitParameters' => '--executor-memory 2G --num-executors 2'
            ]
        ]
    ]);
    echo "Job submitted: " . $result->get('id');
} catch (AwsException $e) {
    echo $e->getMessage();
}
?>

使用JavaScript列出EMR容器作业运行状态

javascript

const { EMRContainersClient, ListJobRunsCommand } = require('@aws-sdk/client-emr-containers');

const client = new EMRContainersClient({
    region: 'eu-central-1',
    credentials: {
        accessKeyId: 'YOUR_API_KEY',
        secretAccessKey: 'YOUR_SECRET_KEY'
    }
});

async function listJobRuns(virtualClusterId) {
    try {
        const command = new ListJobRunsCommand({
            virtualClusterId: virtualClusterId,
            maxResults: 10
        });
        const response = await client.send(command);
        
        response.jobRuns.forEach(job => {
            console.log(`Job ID: ${job.id}, State: ${job.state}, Created: ${job.createdAt}`);
        });
    } catch (error) {
        console.error('Error listing job runs:', error);
    }
}

// 使用示例
listJobRuns('your-virtual-cluster-id');

常见问题

什么是Amazon EMR on EKS？

Amazon EMR on EKS是一种部署选项，允许您在Amazon Elastic Kubernetes Service（EKS）上运行开源大数据框架（如Apache Spark）。它使您能够使用Amazon EMR的运行分析应用程序的体验，同时利用EKS的容器编排能力，无需管理底层基础设施。

使用Amazon EMR Containers API需要哪些权限？

要使用Amazon EMR Containers API，您需要具有适当的IAM权限。通常需要以下权限：emr-containers:CreateVirtualCluster、emr-containers:CreateJobRun、emr-containers:ListJobRuns等。此外，还需要访问相关EKS集群和S3存储桶的权限。建议使用最小权限原则配置IAM策略。

如何监控EMR容器作业的运行状态？

您可以通过多种方式监控EMR容器作业：1) 使用DescribeJobRun API获取详细状态信息；2) 查看CloudWatch日志和指标；3) 通过EKS集群的Kubernetes仪表板监控Pod状态；4) 使用Amazon Managed Service for Prometheus和Grafana进行高级监控。作业状态包括PENDING、SUBMITTED、RUNNING、FAILED、CANCELLED等。