Pipelines PGFS with S3-compatible storage

PGFS uses the s3: prefix to indicate an S3-compatible storage system. You can use the s3: prefix to access files on an S3-compatible storage system.

Using S3-compatible storage through PGFS with SQL

For example:

CREATE SERVER images_s3 FOREIGN DATA WRAPPER pgfs OPTIONS (url 's3://my-bucket', , skip_signature 'true');

This command creates a server object called images_s3 that connects to the S3-compatible storage system at my-bucket. With no credentials provided, the server will assume the S3 storage system is open to the public. For a public bucket, ensure that you include the skip_signature option and set it to true

You can then create a foreign table that is associated with the server:

CREATE FOREIGN TABLE images (id INT, name TEXT) SERVER images_s3;

This foreign table is associated with the images_s3 server object and can be used to access the files in the my-bucket directory.

You can also use the s3: prefix to access files on the S3-compatible storage system when you create a foreign table. For example:

CREATE FOREIGN TABLE images (id INT, name TEXT) SERVER images_s3 OPTIONS (filename 's3://my-bucket/image1.jpg');

This foreign table is associated with the images_s3 server object and can be used to access the file my-bucket/image1.jpg.

Supported options with S3-compatible storage

The following options are supported when you create a storage location with S3-compatible storage: