blob_storage

prefect_azure.blob_storage

Integrations for interacting with Azure Blob Storage

Functions

blob_storage_download

blob_storage_download(container: str, blob: str, blob_storage_credentials: 'AzureBlobStorageCredentials') -> bytes

Downloads a blob with a given key from a given Blob Storage container. Args: blob: Name of the blob within this container to retrieve. container: Name of the Blob Storage container to retrieve from. blob_storage_credentials: Credentials to use for authentication with Azure. Returns: A bytes representation of the downloaded blob. Example: Download a file from a Blob Storage container

from prefect import flow

from prefect_azure import AzureBlobStorageCredentials
from prefect_azure.blob_storage import blob_storage_download

@flow
def example_blob_storage_download_flow():
    connection_string = "connection_string"
    blob_storage_credentials = AzureBlobStorageCredentials(
        connection_string=connection_string,
    )
    data = blob_storage_download(
        container="prefect",
        blob="prefect.txt",
        blob_storage_credentials=blob_storage_credentials,
    )
    return data

example_blob_storage_download_flow()

blob_storage_upload

blob_storage_upload(data: bytes, container: str, blob_storage_credentials: 'AzureBlobStorageCredentials', blob: Optional[str] = None, overwrite: bool = False) -> str

Uploads data to an Blob Storage container. Args: data: Bytes representation of data to upload to Blob Storage. container: Name of the Blob Storage container to upload to. blob_storage_credentials: Credentials to use for authentication with Azure. blob: Name of the blob within this container to retrieve. overwrite: If True, an existing blob with the same name will be overwritten. Defaults to False and an error will be thrown if the blob already exists. Returns: The blob name of the uploaded object Example: Read and upload a file to a Blob Storage container

from prefect import flow

from prefect_azure import AzureBlobStorageCredentials
from prefect_azure.blob_storage import blob_storage_upload

@flow
def example_blob_storage_upload_flow():
    connection_string = "connection_string"
    blob_storage_credentials = AzureBlobStorageCredentials(
        connection_string=connection_string,
    )
    with open("data.csv", "rb") as f:
        blob = blob_storage_upload(
            data=f.read(),
            container="container",
            blob="data.csv",
            blob_storage_credentials=blob_storage_credentials,
            overwrite=False,
        )
    return blob

example_blob_storage_upload_flow()

blob_storage_list

blob_storage_list(container: str, blob_storage_credentials: 'AzureBlobStorageCredentials', name_starts_with: Optional[str] = None, include: Union[str, List[str], None] = None, **kwargs) -> List['BlobProperties']

List objects from a given Blob Storage container. Args: container: Name of the Blob Storage container to retrieve from. blob_storage_credentials: Credentials to use for authentication with Azure. name_starts_with: Filters the results to return only blobs whose names begin with the specified prefix. include: Specifies one or more additional datasets to include in the response. Options include: ‘snapshots’, ‘metadata’, ‘uncommittedblobs’, ‘copy’, ‘deleted’, ‘deletedwithversions’, ‘tags’, ‘versions’, ‘immutabilitypolicy’, ‘legalhold’. **kwargs: Additional kwargs passed to ContainerClient.list_blobs() Returns: A list of dicts containing metadata about the blob. Example:

from prefect import flow

from prefect_azure import AzureBlobStorageCredentials
from prefect_azure.blob_storage import blob_storage_list

@flow
def example_blob_storage_list_flow():
    connection_string = "connection_string"
    blob_storage_credentials = AzureBlobStorageCredentials(
        connection_string="connection_string",
    )
    data = blob_storage_list(
        container="container",
        blob_storage_credentials=blob_storage_credentials,
    )
    return data

example_blob_storage_list_flow()

Classes

AzureBlobStorageContainer

Represents a container in Azure Blob Storage.

This class provides methods for downloading and uploading files and folders to and from the Azure Blob Storage container.

Attributes:

• container_name: The name of the Azure Blob Storage container.
• credentials: The credentials to use for authentication with Azure.
• base_folder: A base path to a folder within the container to use for reading and writing objects.

Methods:

download_folder_to_path

download_folder_to_path(self, from_folder: str, to_folder: Union[str, Path], **download_kwargs: Dict[str, Any]) -> Coroutine[Any, Any, Path]

Download a folder from the container to a local path.

Args:

• from_folder: The folder path in the container to download.
• to_folder: The local path to download the folder to.
• **download_kwargs: Additional keyword arguments passed into BlobClient.download_blob.

Returns:

• The local path where the folder was downloaded.

download_object_to_file_object

download_object_to_file_object(self, from_path: str, to_file_object: BinaryIO, **download_kwargs: Dict[str, Any]) -> Coroutine[Any, Any, BinaryIO]

Downloads an object from the container to a file object.

Args:

• from_path : The path of the object to download within the container.
• to_file_object: The file object to download the object to.
• **download_kwargs: Additional keyword arguments for the download operation.

Returns:

• The file object that the object was downloaded to.

download_object_to_path

download_object_to_path(self, from_path: str, to_path: Union[str, Path], **download_kwargs: Dict[str, Any]) -> Coroutine[Any, Any, Path]

Downloads an object from a container to a specified path.

Args:

• from_path: The path of the object in the container.
• to_path: The path where the object will be downloaded to.
• **download_kwargs: Additional keyword arguments for the download operation.

Returns:

• The path where the object was downloaded to.

get_directory

get_directory(self, from_path: Optional[str] = None, local_path: Optional[str] = None) -> None

Downloads the contents of a direry from the blob storage to a local path.

Used to enable flow code storage for deployments.

Args:

• from_path: The path of the directory in the blob storage.
• local_path: The local path where the directory will be downloaded.

list_blobs

list_blobs(self) -> List[str]

Lists blobs available within the specified Azure container.

Used to introspect your containers.

Returns:

• A list of the blobs within your container.

put_directory

put_directory(self, local_path: Optional[str] = None, to_path: Optional[str] = None, ignore_file: Optional[str] = None) -> None

Uploads a directory to the blob storage.

Used to enable flow code storage for deployments.

Args:

• local_path: The local path of the directory to upload. Defaults to current directory.
• to_path: The destination path in the blob storage. Defaults to root directory.
• ignore_file: The path to a file containing patterns to ignore during upload.

read_path

read_path(self, path: str) -> bytes

Reads the contents of a file at the specified path and returns it as bytes.

Used to enable results storage.

Args:

• path: The path of the file to read.

Returns:

• The contents of the file as bytes.

upload_from_file_object

upload_from_file_object(self, from_file_object: BinaryIO, to_path: str, **upload_kwargs: Dict[str, Any]) -> Coroutine[Any, Any, str]

Uploads an object from a file object to the specified path in the blob storage container.

Args:

• from_file_object: The file object to upload.
• to_path: The path in the blob storage container to upload the object to.
• **upload_kwargs: Additional keyword arguments to pass to the upload_blob method.

Returns:

• The path where the object was uploaded to.

upload_from_folder

upload_from_folder(self, from_folder: Union[str, Path], to_folder: str, **upload_kwargs: Dict[str, Any]) -> Coroutine[Any, Any, str]

Uploads files from a local folder to a specified folder in the Azure Blob Storage container.

Args:

• from_folder: The path to the local folder containing the files to upload.
• to_folder: The destination folder in the Azure Blob Storage container.
• **upload_kwargs: Additional keyword arguments to pass to the upload_blob method.

Returns:

• The full path of the destination folder in the container.

upload_from_path

upload_from_path(self, from_path: Union[str, Path], to_path: str, **upload_kwargs: Dict[str, Any]) -> Coroutine[Any, Any, str]

Uploads an object from a local path to the specified destination path in the blob storage container.

Args:

• from_path: The local path of the object to upload.
• to_path: The destination path in the blob storage container.
• **upload_kwargs: Additional keyword arguments to pass to the upload_blob method.

Returns:

• The destination path in the blob storage container.

write_path

write_path(self, path: str, content: bytes) -> None

Writes the content to the specified path in the blob storage.

Used to enable results storage.

Args:

• path: The path where the content will be written.
• content: The content to be written.