Eliminate Unnecessary Duplication of Work and Data

The aim here is to avoid redundant effort.

Begin by ensuring you're performing incremental loading for all data sources. While some sources may not offer Change Data Capture (CDC) capabilities, you might devise innovative ways to pull only mutated data.

One method involves generating a hash of all values per ID at both the source and destination, comparing hash values, and copying only the mutated values.

Consider performing frequent incremental loads only for frequently used data. Then, implement an "eventual consistency" or "healing" method that performs full loads less frequently, such as weekly or monthly.

Next, assess how many copies of the data you keep and how many you actually use. I typically follow a pattern of keeping one full verbatim copy, one full transformed copy (including data type transformations), and other mutations like data marts. Anything outside these categories is expendable and should be minimized.

Evaluate Your Logging Practices

Over-retaining logs is another potential pitfall. Most AWS services have a configurable log destination. Use this to monitor what's being logged and set up lifecycle rules. If you need to retain logs for years due to compliance requirements, S3's intelligent tiering is a feature I highly recommend.

Lastly, be aware that some services, like Glue, use S3 as swap storage. If you're copying large amounts of data, you might also be keeping a copy among your logs. While you can't override this behavior, consider reducing the data loads instead.

Leverage Capacity Reservations and Lower Execution Tiers

Data teams often work on long-term tasks, so capacity reservations make sense for long-term cost control. You pre-purchase what you're sure to be using a year from now. For example, with Redshift, you're unlikely to stop using it or reduce the number of nodes within the next year. So, reserving that capacity can reduce your bill by 25% in less than 10 minutes.

Another tip: Glue offers two execution tiers - standard and flex. While the official guideline suggests using flex only for development workloads, I've successfully used it in production without issues. Using job retries in combination with flex execution has proven reliable in my experience.

Remember, AWS cost optimization is about making informed decisions that align with your business objectives and budget constraints. By planning and executing these changes, you're well on your way to mastering AWS cost optimization.

Establish the strengths and weaknesses of your team

In the quest for AWS cost optimization, understanding your team's strengths and weaknesses is the crucial first step. The composition of your team, their skills, and their experience can significantly influence your cloud strategy and, by extension, your AWS costs. You may have a senior data architect at the helm, ideally in a team lead or managerial role. This individual's expertise can be instrumental in determining the optimal setup given your current limitations, data volume, velocity, and variety. The questions you need to address could range from whether you require a data lake or a standard data warehouse, to whether you should use AWS native technology or simply use AWS as a hosting service. On the other hand, your team might comprise members with a strong DevOps background. In such a scenario, you could lean towards a fully automated CI/CD and deploy infrastructure-as-code. This approach can facilitate rapid iteration from proof-of-concepts (PoCs) to establishing a more cost-efficient cloud strategy. If your team consists of developers or SQL wizards, they might prefer to tackle complex issues with code versus SQL procedures. This preference can significantly impact your cost sheet. Are you leveraging the best tools to augment your team's skills? Regardless of your team's structure and strengths, the key is to harness them effectively. Tailor your processes and tools to capitalize on your team's unique edge. By doing so, you not only enhance your team's performance but also potentially lower your running costs. Remember, AWS cost optimization is not a one-size-fits-all strategy. It's a journey that requires a deep understanding of your team's capabilities and a willingness to adapt and evolve. By leveraging your team's strengths, you take the first step towards mastering the art of AWS cost optimization.

Take a good, long look at your data & streams

The journey towards AWS cost optimization continues with a thorough examination of your data and streams. This step involves prioritizing how your data is used and provisioned, which can significantly impact your AWS costs. Start by asking yourself: What are your data needs? Do you require streaming services, or would batch processing suffice for your stakeholders? How frequently does your data need to converge - once a day, an hour, or a minute? Is all your data business-critical at all times, or are there certain data flows that are rarely, if ever, accessed? How many copies of your data do you need to maintain? What would be the impact if your data were unavailable for an hour, a day, or even permanently? It's important to note that the answers to these questions are rarely static. They evolve over time, influenced by changes in business needs, stakeholder requirements, and even pricing models. Therefore, it's advisable to revisit these questions periodically - a good rule of thumb is to do a comprehensive review at least once a year. Remember, understanding your data landscape is not just about identifying what you have, but also about understanding how it's used, by whom, and when. By gaining a deep understanding of your data and streams, you can make informed decisions that align with your business objectives and budget constraints, taking you one step closer to mastering AWS cost optimization.

Analyze your bill

The path to AWS cost optimization is paved with a deep understanding of your expenditure. At the highest level, your AWS bill typically comprises four components: storage (the cheapest component, scales linearly), compute (processor), memory, and I/O (traffic, usually scales exponentially). While AWS's console-provided Cost Management is a good starting point, a more granular understanding of your costs requires a deeper dive. I recommend using AWS CLI to extract data, which you can then persist and manually enrich with retail prices. For instance, you might know the DUI price for AWS Glue, but the costs of individual workflows or jobs remain opaque. You might be aware of the price per GB of storage in S3, but the daily I/O volume might be unknown. And let's not forget the additional services like Config, Cloud Trail, Cloud Watch, Guard Duty, KMS, or even Macie that need to be factored in per GB of storage. My suggested approach is to focus on the top 10 most expensive services and dissect each line item. More often than not, you'll find that the costs don't add up, indicating that the service is being used for something you haven't factored in. It's crucial to know exactly where your money is going. Armed with this knowledge and a clear understanding of your data requirements, you'll be in a position to propose immediate cost optimization tactics or an overarching strategy to re-orchestrate data flows. This can better serve your stakeholders' needs while simultaneously lowering costs. However, don't forget to factor in the effort required to realize this vision! Remember, AWS cost optimization is not just about cutting costs; it's about making informed decisions that align with your business objectives and budget constraints. By decoding your AWS bill, you take another step towards mastering AWS cost optimization.

Challenge

Now for the true challenge: writing is as slow as your network, which is usually 300-500ms per transaction. Multiply that RTT (round-trip time), add an occasional replay (write error, requiring resending of the data) and we're looking at...

4.3B * 0.33s = 16 570 days ???

Not if we can help it.

Hack #1: documentation aka. RTFM

Azure table supports transactional writes, under the following limitations:

• batches are 100 records (or less) long

• batches contain the same partition key

Under perfect conditions, this lowers our effort to 165 days. Definitely a 3.6 result.

So we need to work harder.

Approach

By leveraging the generation approach described earlier, the time to generate a /8 network I've achieved (32 core Xeon w. 64G RAM) is an average of 23s. [2] We'll use the /8 as building blocks for our optimization task.

Now, a /8 has 1.6M records, meaning the 65536 partitions we're filling would have around 256 records - enough for 3 requests to the Azure Table.

The process in a nutshell looks like this:

0. SHA-256 records are generated for a /8 network in a Pandas DataFrame.

1. The DataFrame is filtered by the common PartitionKey.

2. The records are prepared and transmitted in parallel.

At this point, we perceive (2) as the slowest process. Why don't we send more records in parallel?

Consider this: a /8 takes around 5.5-6 GB of RAM. Based on your hardware, you may decide that generating 2* /8 dataframes, appending them as one is more efficient (thus filling the 65536 partitions with ~512 records, for ~6 requests). But soon you'll realise that (1) gets significantly slower... around the point you fill 1/2 of the physical memory of your machine, it starts really dragging.

Obviously, unless you use dask (which we will explore in our next post), Spark, MapReduce or any similar alternative, (1) is synchronous and slow. So, it's up to you to measure and optimize.

One more detail is also worth noting: you probably noticed that I used "around" for the number of records. Indeed, due to natural distribution, you might encounter partitions with less than 220 records or more than 270. These fluctuations, also due to normal distribution, get smaller the more records you add to the partitions. Therefore, if you generate 3* /8 networks, you're almost guaranteed you'll need exactly 8 transactions to Azure. Another observation: if you go with 2* /8 networks, the last request is a little wasteful, as you're only sending ~12 records through (out of ~512).

Code

To parallize requests to Azure, it's most efficient to use Threads rather than Processes. Remember, these would be IDLE most of the time:

def transact(operations : list):
    table_client.submit_transaction(
                    operations
                )

def split_and_submit(partition : list):
    operations = []
    tpool = ThreadPool(processes = 32)
    for record in partition:
        operations.append(('upsert', record))
        if len(operations) == 100:
            rq = tpool.map_async(transact, [operations])
            operations = []
            rq.get()
    if len(operations) > 0:
        rq = tpool.map_async(transact, [operations])
        rq.get()
    tpool.close()

The above setup will create threads for each group of 100 records. Now you can (and should) run ThreadPools in parallel Processes too:

alpha = generate(20)

alpha.index = alpha["PartitionKey"].copy()
alpha.index.name = None
iterator = list(alpha["PartitionKey"].unique())

processed = return_temp('processed.temp')
unprocessed = [i for i in iterator if i not in processed]
alpha = alpha.drop(processed)

for chunk in chunker(unprocessed, int(CORES)):
    old = timer()
    print("chunk", end=' ')
    a = alpha.loc[chunk]
    print("located in ", round(timer()-old, 2), "s;", sep = '', end=' ')
    with mp.Pool(CORES) as pool:
        while True:
            try:
                r = pool.map_async(
                    split_and_submit,
                    [(a.loc[c]).to_dict('records') for c in chunk]
                    )
                r.get()
            except:
                continue
            break
    pool.join()
    pool.close()
    print("submitted to Azure in ", round(timer()-old, 2), "s;", sep = '', end=' ')
    processed.extend(chunk)
    with open("processed.temp", 'w') as file:
        writer = csv.writer(file)
        writer.writerow(processed)
    print("and finalized in ", round(timer()-old, 2), "s; progress:", round(int(len(processed)/32)/20.48, 2), '%', sep = '')

The code above uses a file - "processed.temp", to persist the processed partitions in case of errors. As the main process runs for hours and is error prone, it makes sense to track your progress.

Altenatively, you can use:

• ThreadPools instead of MultiprocessPools that call ThreadPool. That's tricky and will be more unstable - and eventually the instability translates to time losses.

• A CPU-rich GPU for the parallel processing

• A ThreadPool with dynamic feed of partitions. The tricky part here is that as part (1) above is slow, so you need to filter and split partitions in mp.Pools to feed the ThreadPool