Making Calcapp perform better with very large apps has been an ongoing project for us for several years now. In 2018, we started compressing apps sent from our server. Later that year, we made navigating such apps much faster and improved the way our server responds to app requests. In 2019, we responded to an influx of users by tripling the memory available to our server, making very large apps run much better.
Our new release further improves our handling of very large apps. When users download such apps, they will find that they download much faster.
For small apps, you probably won’t notice much difference. For very large apps, however, the savings can be huge. We managed to shrink one app from 158 MB to 30 MB (or from 8.9 MB to 1.7 MB with compression turned on).
Such a large app is not only downloaded faster, it is also processed faster when it starts. Also, storing it for offline use doesn’t consume as much storage for your users.
This post is somewhat technical. If you’re not interested in the nitty-gritty, you can safely skip this material, knowing that large apps now work better.
Omitting default values
One technique we used to make apps download faster is that we no longer send “default values.” What that means is that if you add 100 fields to your app, we used to store all values you set in the inspector, even if you never changed the defaults, for all 100 fields.
Number fields are normally formatted with two decimal places, and if you don’t change that setting, we will no longer send that information when an app is downloaded — your app will just assume that number fields should be formatted with two decimal places unless if hears differently.
It turns out that there are lots of values that app authors rarely change, meaning that we can leave them out most of the time. This makes for significant savings.
Making information about your app easily accessible
There is some information about your apps that our server needs to process very often. One of the most important pieces of information is what version of Calcapp is targeted.
From an app author’s perspective, there are no Calcapp versions, it just regularly improves. Behind the scenes, though, we make changes all the time. We make changes to the way your apps are stored in our database and we make changes to how your apps are represented internally. We even rename properties and functions from time to time, which necessitates rewriting your formulas (prior to this release, for instance, the form screen property NextScreenAvailable was named NextPanelAvailable).
In fact, we spend lots of time writing software that you never see, whose only purpose is to move your apps from one Calcapp version to a more current version, which we refer to as our “migration process.” We have spent a lot of time optimizing this process, with a view towards minimizing server downtime. It now takes roughly 24 hours to migrate all your apps to a new version, but our server remains available during all that time, except for around ten minutes when you can’t change any apps (you can still run apps).
The Calcapp version is really important, and it should be stored in a way that makes it fast to access. We also need quick access your decimal separator preference (decimal comma or decimal point?), so that we can convert all the formulas to your chosen decimal separator when necessary.
Collectively, all this “meta” information is stored in your app “header.” It’s supposed to be at the very start of your app (hence the name), but for various technical reasons, it was sometimes located in the middle, or at the very end of your app. That meant that for very large apps, our server had to repeatedly search through hundreds of megabytes of data to locate the information it needed, making it sluggish.
Thankfully, we have now fixed this issue, meaning that large apps should no longer slow down the server in this way.
Compressing dependencies
Our server does a lot of heavy lifting behind the scenes to ensure that your apps run as quickly as possible. One thing it does is that it figures out what properties depend on one another. Let’s say that the value of Field1 is defined as follows:
When the value of Field2 changes, your app knows that the value of Field1 depends on it and that it needs to be re-calculated, but that it can use old (cached) values for unaffected properties. We say that Field2.ValueField2,Value is a dependency of Field1.ValueField1,Value and that Field1.ValueField1,Value is a dependent of Field2.ValueField2,Value. This makes your app run much faster than if it had to re-calculate everything every time anything changed.
(Have you ever used “trace precedents” in a spreadsheet? It’s the same idea.)
This information used to take up a lot of space. When Field2.ValueField2,Value noted that Field1.ValueField1,Value depended on it, it used the full name of the screen it belonged to, the full name of Field1 and the full name of the Value property. That can take up a lot of space. Now, we assign numerical identifiers to properties (like “232”) and just reference those.
We also stored even indirect dependents (say, if Field3.ValueField3,Value depends on Field2.ValueField2,Value, which depends on Field1.ValueField1,Value, Field1.ValueField1,Value would store both Field2.ValueField2,Value and Field3.ValueField3,Value as dependents). For some large apps, a single key field could have thousands or tens of thousands of indirect dependents. Just storing the dependents of fields could take up hundreds of megabytes of storage for particularly large apps, which is data that would need to be sent and read every time your app was downloaded.
We now only store direct dependents, meaning that your apps need to do a bit more work when run, but it’s well worth it to save a lot of space.
Also, there are a lot of what we call implicit dependencies between properties. The FormattedValue property of number fields has an implicit dependency on the Value property, meaning that even if you refer to a formatted value, your formula will get re-calculated when the value changes.
These implicit dependencies used to be communicated explicitly by our server. Now, we don’t do that anymore, because your apps are smart enough to know about all those implicit dependencies. That also saves a lot of space.
Finally, we no longer reference properties by name in the JavaScript expressions that your Calcapp formulas are converted to by our server. Instead we use — you guessed it — the numerical identifier we now generate. That makes the JavaScript versions of your formulas a lot shorter too.
The road ahead for large apps
We’ve worked hard for years now on improving support for large apps, and we’ll be the first to admit that we still have a lot of work to do.
In the past, we used to advertise our White Label plan as supporting arbitrarily large apps. We no longer do that, because Calcapp can’t handle arbitrarily large apps. We now advertise our best plan as supporting “up to 5,000 fields.” We don’t actually enforce that limit — you’re welcome to go over it if you like — but we can’t in good conscience claim that we do a good job of supporting apps larger than that.
We actually do pretty well, compared to other app builders. Microsoft asks that you don’t add more than 500 controls to apps built with PowerApps, for instance.
We would like to do better, though, and we have a few changes in mind that we want to make part of our next-generation Calcapp 4 project. Stay tuned, we’ll have more to say on this subject in the future.
What you can do to make large apps run better today
One way to make large apps run better today is to not make them as large — at least not as perceived by Calcapp. In the past, many app authors have liberally used copy and paste to duplicate screens, where the duplicates often differ very little from one another.
There are several disadvantages to this approach. From an author’s perspective, making changes to the duplicated screens becomes an arduous process, with each change having to be made multiple times. From Calcapp’s perspective, having screens be duplicated many times makes apps very large, which may make them run sluggishly or not at all.
Fortunately, there is now a better way. The new NextScreen property of form screens, text screens and navigators enables the same screen to be reachable from many different places. You can even use conditional logic to determine which screen the user should be taken to.
If you make a number of list screen navigators all take the user to the same screen, use the ActiveNavigator property of list screens to learn which navigator was selected. You can even plug the label of the selected navigator into an XLOOKUP formula to look up a value based on the selection.
To learn more, read our post about the new properties.