Why a Kentico MVC Upgrade Should Not be One Hour of Work
Introduction
Recently, I have been involved in multiple Kentico MVC Upgrades and Hotfixes for versions 10, 11, and 12 of Kentico. Some have went really well, and some have went not so well. This hasn't really surprised me as, this is typical of any complex software upgrade. Raise your hand if you have even seen a Visual Studio or MS Office installation completely crap out due to user permissions, mis-matched .Net framework components, or even in the old days dll hell. Yep, I've been there too.
But here's the deal, sometimes customers, and even developers, expect an upgrade to hit all three points on the magic business triangle of the speed: fast, the cost: cheap, and quality: good. Over the many years of working with Kentico and end clients, I have been told by Digital Marketers, IT Directors, and CEO's that why can't this upgrade "just" be finished tomorrow? Why is it so hard? Can't it "just" be done in an hour or two?
Pro Tip: Just is the most evilest word in I.T. Can't we just add this feature? Can't we just deploy right now? Can't the code just work as I told showed you in my napkin drawing? If I had nickel for every time someone used the word Just…I'd have a lot of nickels.
My answer to all of these questions never beats around the bush. I always look the person directly in the eye, take a small deep breath and simply say "No"
Background
While an upgrade or hotfix is not just a click of a button, I would say that overall the experience of going through an upgrade or hotfix has definitely gotten better over the years. Not perfect by any means, but better. However, it is not fully clear to end clients what an upgrade actually means in terms of effort. I'm hoping though that you don't have to end of writing emails like this your clients:
…I'm sorry for the confusion. But we are unfortunately not on the same page here. We might need to jump on a phone call on Monday to go over it in more detail. However, any upgrade that we do for any of our customers, whether it is Kentico software, SQL Server software, asp.net framework version, or even custom software that we have written each line of code, is not something that can be done in "just an hour or two"….
Once I get that customer or developer on a call, I tend to start my explanation of what a Kentico Upgrade is by not talking about Kentico. I try to use examples that might be familiar to anyone like the following. Imagine if John Doe buys a copy of Microsoft Office 2018 at Best Buy for his laptop. When Office 2019 comes out it is the customer's choice to purchase and install the upgrade, not the business they purchased it from. Would it be on Best Buy because they sold the original copy? Would it be on Microsoft because they created the software? It really is up to John to pay for that upgrade at the end of the day. And if that upgrade has an error, and it takes 3 times to re-run the upgrade wizard because John didn't have the right security permissions on his installation folder, that time that it takes to do the upgrade, is that up to Best Buy or Microsoft to cover? No, it's up to John.
Now, let's say BizStream is upgrading to Kentico 12 MVC from Kentico 11 MVC for a customer named Travel Rock Stars. There is not much difference in the situation outlined above compared to one we would have with Travel Rock Stars. Just change out Microsoft with Kentico, Best Buy with BizStream, and John for Travel Rock Stars. To both Kentico and BizStream an upgrade is a cost for our clients of owning and maintaining a custom software application. I'm pretty sure other vendors would see it this way as well.
I think this analogy makes sense when explaining why upgrades are up to end clients and not the software providers. But we haven't touched the amount of effort yet. Let's do that next.
So Why Does it Take So Long
I do want to try to clear up why the cost/effort is as much as it is, because I think that is playing a part here too. When we say "upgrade", BizStream knows what this entails, because we have probably done 50+ upgrades over the years from many different Kentico versions. But in our fictional scenario, this would probably be Travel Rock Star's first upgrade, I think it makes sense to show them the details of what needs to happen because an upgrade is not just one step.
Not Just One Step
It is considered a best practice to not run on the .0 version of Kentico software. So really, any upgrade is two upgrades right out of the gate. The 11.0.x to 12.0 upgrade is one unit of work. We need to perform that first main 12.0 upgrade, and then right after that, perform what Kentico calls a Hotfix to get to a more stable version, which is a second unit of work. For example, after 12.0 first came out, we waited a few weeks and eventually recommended hotfixing to version 12.0.6 for any 12.0 project. This was due to the fact 12.0.6 is a stable release that has a key fix to the MVC Page Builder which is a feature that every marketing team out there is going to want to use. It's the best part of Kentico 12.0.
That means we need to get the Travel Rock Star's codebase updated and working to 12.0 following all of these steps: https://docs.kentico.com/k12/installation/upgrading-to-kentico-12 then right after that, we need to perform the hotfix following these steps: https://docs.kentico.com/k12/installation/hotfix-instructions-kentico-12.
Both sets of instructions portray the amount of work, it involves SQL work, work on the CMS Admin tool upgrade, and work on the MVC live site codebase upgrade. Remember the key Nuget Packages are actually Kentico NuGet packages, not just a few BizStream or third-party packages. In the 12.0 upgrade Kentico has made some larger changes to their NuGet Packages that we need to translate to the live site solution. Kentico calls them out as breaking changes:
All of this work is done on a local machine first, and typically it doesn't go perfect in one run. It might take a few attempts to get it right.
More Than Just One Environment
After the initial local upgrade, you should think about needing to then re-run the process on the shared DEV/QA environment that you have to not lose our changes there either. After those two environments are tested and pass your QA checks, we would then need to wrap it all up, and deliver it to a non production Travel Rock Stars Azure App Service instance for their testing. To do this, we always recommend that you take a backup of a production environment first (both database and code) and use that are your target database for the upgrade. This way you have a realistic amount of real data and content and you can time about how long each step of the upgrade takes.
This is where the first set of "hidden" effort is to most non-technical customers. It can easily take 4 -6 hours of time to back up and restore a production instance of a web site. Because this involves the actual backups, zipping of files, transferring copies of the files down to a developer workstation, restoring the files locally, and then re-configuring the settings to have the correct site domain alias, site licenses, email settings (disabled), web farm setup (disabled) and scheduled tasks (disabled). This is all to just prep for an upgrade. And we just talked about at least 3 environments in play here. The math does start to add up.
Unplanned Things Will Happen
Oh, and speaking of backups, they don't always go so great when you are in an upgrade cycle. We recently had an issue when restoring a BACPAC file from a live Azure SQL database to a local SQL Server. The very last step of the Import data tier application wizard errored on us. With the following weird message:
This one stumped us a bit, and after speaking with Kentico support, it turned out that there were some orphaned rows in the Analytics_* tables inside the Kentico database catalog that were causing the keys / indexes of the database to not recreate correctly on the restore of the database. This is a great example of an item where you don’t plan for time to worry about Foreign Key Constraints not restoring, and won’t have time in your estimate for every possibly thing.
Pro Tip: The way around this issue is to clone the Azure SQL Database into a new temporary database in Azure SQL that is not connected to the live running website. On that clone, you can then delete the offending Analalytics_WeekHits or Analalytics_DayHits, or whichever Analytics_* table rows from the Kentico database before you create the new BACPAC file using the Azure Query Explorer that is in the Azure Portal. Once you have completed that, the restore will work just fine.
This truly is an example of one of those little fun issues that are not normally thought about when quoting out what an upgrade takes. But to get it perfectly, you have to correct these types of things. You can’t “just” ignore them.
Continue on to Part 2
Keep reading the second part of Why a Kentico MVC Upgrade Should Not be One Hour of Work to find out more reasons why.
