The connection program copy
This option is only configurable for Glue version 2.0 and above. "idleTimeBetweenReadsInMs": (Optional) The minimum time delay between twoĬonsecutive getRecords operations, specified in ms. "addIdleTimeBetweenReads": (Optional) Adds a time delay between twoĬonsecutive getRecords operations. "maxRecordPerRead": (Optional) The maximum number of records to fetchįrom the Kinesis data stream in each getRecords operation. "maxFetchRecordsPerShard": (Optional) The maximum number of records toįetch per shard in the Kinesis data stream. To fetch a record from the Kinesis data stream per shard, specified in milliseconds "maxFetchTimeInMs": (Optional) The maximum time spent in the job executor "startingPosition": (Optional) The starting position in the Kinesis data StreamARN (Required) The ARN of the Kinesis data stream. You can use the following methods under the GlueContext object to consume This option is available in AWS Glue version 3.0 only.ĭesignates a connection to Amazon Kinesis Data Streams. The data output will contain an additional column named "headers" with type Array. "includeHeaders": (Optional) Whether to include the Kafka headers. Partitions is equal to the number of Kafka partitions. The default value is null, which means that the number of spark "minPartitions": (Optional) The desired minimum number of partitions to Theĭefault value is null, which means that the consumer reads all offsets until the known Is proportionally split across topicPartitions of different volumes. Of offsets that are processed per trigger interval. "maxOffsetsPerTrigger": (Optional) The rate limit on the maximum number "retryIntervalMs": (Optional) The time in milliseconds to wait before
"numRetries": (Optional) The number of times to retry before failing toįetch Kafka offsets. "pollTimeoutMs": (Optional) The timeout in milliseconds to poll data from
Import .DynamoDbDataSinkĭef main(sysArgs: Array): Unit =. Use the following connection options with "connectionType": "dynamodb" as a "maxBatchSize": (Optional): The maximum batch size for bulk operationsįor sample code, see Examples: Setting Connectionĭesignates a connection to Amazon DynamoDB.Ĭonnection options differ for a source connection and a sink connection. Ifįalse, only fields in the document that match the fields in the dataset "replaceDocument": (Optional) If true, replaces the wholeĭocument when saving datasets that contain an _id field. "extendedBsonTypes": (Optional) If true, allows extendedīSON types when writing data to Amazon DocumentDB. "password": (Required) The Amazon DocumentDB password. "username": (Required) The Amazon DocumentDB user name. "collection": (Required) The Amazon DocumentDB collection to write to. "database": (Required) The Amazon DocumentDB database to write to. "uri": (Required) The Amazon DocumentDB host to write to, formatted as Use the following connection options with "connectionType": "documentdb" as Microsoft SQL Server database (see JDBC connectionType Values)ĭesignates a connection to Amazon DocumentDB (with MongoDB compatibility).Ĭonnection options differ for a source connection and a sink connection. PostgreSQL database (see JDBC connectionType Values)Īmazon Redshift database (see JDBC connectionType Values) Oracle database (see JDBC connectionType Values)įiles stored in Amazon Simple Storage Service (Amazon S3) in the Apache Hiveįiles stored in Amazon S3 in the Apache Parquet file MySQL database (see JDBC connectionType Values)
Kafka or Amazon Managed Streaming for Apache Kafka Spark, Athena, or JDBC data stores (see Custom and AWS Marketplace connectionTypeĪmazon DocumentDB (with MongoDB compatibility) database