Benchmarking MongoDB from 3.6 to 8.0 - Part 2

Introduction

Welcome to part two of this benchmarking series!

If you have landed here and are not fully aware of what is going on, this is a two-part blog post in which I share my results after benchmarking MongoDB from version 3.6 to 8.0.

In this post, I’m not going to describe the main idea behind this series, the environment, or how the tests were conducted. For that, we have the first part, which covers all those points with all the details I could think of.

If you want to get more context and understand further details behind the tests, you can find the link below to that first part:

• Benchmarking MongoDB from 3.6 to 8.0 - Part 1

How to use

Here, it’s a guide about the data and how to use/interpret it.

Below, you will see five tabs that you can access to see the results for each number of threads tested and the scenarios.

• You can just click in, and it will expand with the results.

Once you expand the tab, you will see a table with the operations tested (Scenario) on the far left column, and the result across all versions on the right.

• The numbers behind each version on that row mean Operations per Second.

For example:

• For the scenario of Aggregation.FindProjectionDottedField.Indexed.
  • MongoDB 3.6 was able to perform 549.82 operations per second.
  • MongoDB 4.0 was able to perform 520.79 operations per second.
  • and so on…

For more details on the methodology, please refer to the section that further explains it in Part 1.

To help you with the analysis, the tables hold a few features for that.

1. You can easily compare results by selecting the version of your preference and hovering the mouse over the other release.

For example - Let’s pick version 8.0 as a reference.

• For that, click on the header line:

After that, once you hover your mouse on that same line, it will show the percentage difference when compared to 8.0:

- For that same scenario, MongoDB 7.0 performed 33.19% fewer operations when compared to 8.0.

2. You can enable/disable the “Highlight Version” 

The idea was pretty simple. Since it’s a benchmark test, I wanted not only to compare the results, as previously, but also to see which release won for each scenario. 

When we enable that option, it highlights which version had better performance for each scenario:

3. Last but not least, if you want to study those numbers, you can easily copy and paste the tables into the software of your preference.

Alongside that, Search is also enabled, so you can filter out the operations of your preference for that table to narrow the analysis by having only the matching results.

Benchmark Results

Conclusion

I’ll leave the final judgment to you.

That’s because one of the ideas behind using different numbers of threads was to allow those with limited resources to gain a further understanding of what they can expect from Mongo in that scenario.

It is nearly impossible to draw a conclusion from all those tests and express it in a single observation. So, take your time, examine the data and methodology more closely, and feel free to share your thoughts in the comments section below.

That being said, a few overall considerations can be shared.

• Is it fair to compare MongoDB 8.0 against older releases, such as 3.6, 4.0, and so on?

In my opinion, it’s not.

If we look at the core component, such as the WiredTiger engine, it has changed drastically to accommodate all the new features and hardening around the product that you like; comparing them at this point is like comparing different products.

Still, those changes shouldn’t be used to support performance regressions over new releases. Even though it isn’t fair to compare both products, you are your own benchmark; and one thing that we really know is that users don’t accept being pushed around, especially when it’s related to losing performance.

Per the releases of the blog posts from the engineering team (1, 2, 3), they were aware of that, and the approach has changed for MongoDB 8:

[..] For instance, if any commit would regress a benchmark by more than 5% for our most important workloads, we would revert the commit. However, this threshold does not detect regressions of 0.1%, and there are thousands of commits per release (e.g., more than 9000 in MongoDB 8.0).

During the release of MongoDB 7.0, we started to take this gradual accumulation of performance loss by tiny regressions of release over release regressions seriously, so we changed the rules of the game. We decided we could not ship MongoDB 7.0 unless it at least matched MongoDB 6.0’s performance on the most important benchmarks.

It’s debatable whether 7.0 has the same performance as MongoDB 6.0, either based on the above tests or on personal experience after working with all those releases.

Still, there is one thing that this study and the engineer’s notes have in common 

• MongoDB 8.0 has indeed better performance than its close predecessors.

Please note that this study doesn’t cover areas such as resource consumption, which is a great candidate for a separate study and posts.

But at this point, my personal recommendation is to upgrade to the 8.0 release as soon as possible to take advantage of those improvements.

Another final observation and a fun fact from this study is that if we remove MongoDB 8.0 from the results, we can see that MongoDB 3.6 is fighting for the second most performant release in this benchmark test, having consistent and great results across the board.

Let me know your thoughts in the comments below, or feel free to reach out to me on LinkedIn.

See ya!

References:

1. MongoDB 8.0: Raising the Bar

2. MongoDB 8.0: Eating Our Own Dog Food

3. MongoDB 8.0: Improving Performance, Avoiding Regressions