Use Azure Storage API to create/download/upload blob

This time, I have a need to upload my Jenkins pipeline output to somewhere and provide a link to user for download. There are 2 needs, 1). we need to run a script to upload output to Azure blob, I want it at a minimal cost, i.e. I don’t want to develop .NET or javascript for it and run the shell script should be enough. 2). The script can generate an one day SAS token for user to download and print the download link in console.

That’s the reason I wrote these shell scripts. 1) create container script to create container if that is not existed. 2). upload file to blob and print download link.

Also, it inspired by https://stackoverflow.com/questions/20103258/accessing-azure-blob-storage-using-bash-curl

Let’s have a quick introduce to our Jenkins environment. We have an AKS cluster as worker, all our jobs will be running in a spawned pod. There are some output will be generated and we want user have a method to keep it(so they won’t keep asking us for them...)

For security concerns, there will be a SAS token for them and the file will be removed in 2 days.

First. Click your storage account -> Data management -> Lifecycle management -> Add a rule

Now, you can filter the container or apply the expiry rule to blobs in that container.

Second. Create a container. I’m using a shell script for that because I need to change the container name frequently. StringToSign is formatted based on link: https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key

This string will be encoded with HMAC-SHA256 over UTF-8. Storage account key is needed for the MAC key for encryption.

Script will form the header base on Azure requirement and call the API. Azure storage blob API can be found here.

Here is the function

authorization="SharedKey"

HTTP_METHOD="PUT"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2020-10-02"

# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"

# Build the signature string
canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${AZURE_STORAGE_ACCOUNT}/${AZURE_CONTAINER_NAME}\nrestype:container"

string_to_sign="${HTTP_METHOD}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}"

# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $AZURE_ACCESS_KEY | base64 -d -w0 | xxd -p -c256)"

# Create the HMAC signature for the Authorization header
signature=$(printf  "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | base64 -w0)

authorization_header="Authorization: $authorization $AZURE_STORAGE_ACCOUNT:$signature"
REQUEST_URL="https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/${AZURE_CONTAINER_NAME}?restype=container"

curl -X ${HTTP_METHOD} \
    -H "$x_ms_date_h" \
    -H "$x_ms_version_h" \
    -H "$authorization_header" \
    -H "Content-Length: 0" \
    ${REQUEST_URL}

Third. Create blob and print SAS token with download link. In this part, besides to call the related API to upload blob. I will also have a need to generate SAS token.

You may read the Azure blob API document to find upload blob API and method to use it. Or you can refer to my full script. I will only share the function about how to generate SAS token.

The SAS token official document is here. Below function is to generate the sig base on the blob path. So the user only has read access in the window on certain blob.

generatesig() {
#######
# From: https://docs.microsoft.com/en-us/rest/api/storageservices/create-account-sas?redirectedfrom=MSDN
#
# StringToSign = signedPermissions + "\n" +  
#                signedStart + "\n" +  
#                signedExpiry + "\n" +  
#                canonicalizedResource + "\n" +  
#                signedIdentifier + "\n" +  
#                signedIP + "\n" +  
#                signedProtocol + "\n" +  
#                signedVersion + "\n" +  
#                signedResource + "\n"
#                signedSnapshotTime + "\n" +
#                rscc + "\n" +  
#                rscd + "\n" +  
#                rsce + "\n" +  
#                rscl + "\n" +  
#                rsct  
    local ST=$(TZ=GMT date -u +"%Y-%m-%dT%H:%M:%SZ" -d "15 mins ago")
    local SE=$(TZ=GMT date -u +"%Y-%m-%dT%H:%M:%SZ" -d "next day")
    local canonicalizedResource="/blob/${AZURE_STORAGE_ACCOUNT}/${AZURE_CONTAINER_NAME}/${FILE_NAME}"
    # StringToSign="${AZURE_STORAGE_ACCOUNT}\nr\nb\no\n${ST}\n${SE}\n\nhttps\n2020-08-04"
    # decoded_hex_key="$(echo -n $AZURE_ACCESS_KEY | base64 -d -w0 | xxd -p -c256)"
    # sig=$(printf  "$StringToSign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | base64 -w0)

    StringToSign="r\n${ST}\n${SE}\n${canonicalizedResource}\n\n\nhttps\n2020-08-04\nb\n\n\n\n\n\n"
    echo ${StringToSign}
    decoded_hex_key="$(echo -n $AZURE_ACCESS_KEY | base64 -d -w0 | xxd -p -c256)"
    sig=$(printf  "$StringToSign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | base64 -w0)

    echo "${OUTPUT_FILE}?sp=r&st=${ST}&se=${SE}&spr=https&sv=2020-08-04&sr=b&sig=${sig}" 
}

Full scripts are here:

trtoolbox/azure_storage at master · tarrantro/trtoolbox

Today is Mid-Autumn Festival holiday, Hope you enjoy your weekend!