Storage Pointers in Solidity

“Here be dragons”

Photo by mauRÍCIO santos on Unsplash

This post is a warning.

Immutability is a two-edged sword. On the one hand, everyone is assured that software will execute as written because no departures from the deployed code will be permitted. On the other hand, immutability implies a very unforgiving environment. This is one of the reasons for adopting a minimalist approach to contract design.

Implicitly, minimalism means contract functions should be self-explanatory and easy to reason about. It’s important that code works and it’s important that observers can see that it works.

This is why storage pointers as a “feature” of the language make me uncomfortable. They can be created unintentionally, they are easy to overlook and the consequences of overlooking them can be catastrophic. Mapmakers of old would warn adventurers about treacherous regions where strange and dangerous and unexpected things can and do happen. They had a short-form expression for mysterious mortal dangers: “Here be dragons.”

What are Storage Pointers?

Solidity has memory variables that don’t persist beyond the execution of a given function, and storage variables that are part of the contract’s persistent state.

Here’s a simple (and safe) example:

contract Safe {
 
 uint x = 100;
 
 function getXAndY() public view returns(uint, uint) {
   uint y = 101;
   return (x,y);
 }
}

So far, so good. We get (100,101) as expected.

There is more going on here, implicitly, than meets the eye. For now, let’s just say “x” is a storage variable and “y” is a memory variable. It’s implied because “x” was declared globally and “y” was declared inside a function.

“x” and “y” are both “scalar” variables. If you’re not familiar with the term, it just means they are of a basic type (in this case, an unsigned integer) with a single value. This is going to be important, because the rules that apply to scalar variables with regard to this issue of memory and storage are different than the rules that apply to indexed variables (arrays and mappings) and structs. That can lead to …

Surprise!

Have a look at this:

contract FirstSurprise {
 
 struct Camper {
   bool isHappy;
 }
 
 mapping(uint => Camper) public campers;
 
 function setHappy(uint index) public {
   campers[index].isHappy = true;
 }
 function surpriseOne(uint index) public {
   Camper c = campers[index];
   c.isHappy = false;
 }
}

Suppose we “setHappy(0)” then invoke “surpriseOne(0)”. Do you suppose campers[0].isHappy? You can be forgiven if you think so, but it is overwritten by

c.isHappy = false;

What’s going on?

“c” is a struct, and we’re allowed to declare them as “memory” or “storage” pointers. The default is storage, so it’s a storage pointer. We said it equals a “campers” at a certain index. Now, it might be inefficient to copy a whole struct from storage to memory, so Solidity just gives “c” a “storage pointer” to wherever the values actually reside. Great, but then we’re at risk of accidentally overwriting storage, which we did when we set one of c’s members to something new.

I am not big fan of this. In my opinion, surprise is an anti-feature in an unforgiving environment.

In fairness, the compiler has improved from early days and there are warnings. For example:

Variable is declared as storage pointer. Use explicit “storage” keyword to silence this warning.

This is a warning to let you know that you are declaring a struct, array or mapping (which you should not do) in a function and it’s going to implicitly be a “storage” (pointer) and you might not be aware of it.

What if you follow the suggestion?

Camper storage c;

That will silence the first warning and give you something else:

Uninitialized storage pointer.

What does that mean? That means it’s a storage pointer that doesn’t know what it should point to because it wasn’t set to anything with “= expression.” So, it will just point to slot 0. That means if you set “c” to anything, that will overwrite whatever is in slot 0. Yikes! This reminds me of IT Haiku circulars that went around before memes caught on. Here’s mine, for entertainment purposes:

Something in slot 0.
Probably important.
Now, it is gone.

Did I mention I do not like this?

Have a look here for another example of how this can and does create quite unexpected results: https://ethereum.stackexchange.com/questions/62384/bytes-variables-are-connected/62394#62394

Another surprise

Here’s the same contract with an added uint variable that is set to 100, and another surprise:

contract AnotherSurprise {
 
 struct Camper {
   bool isHappy;
 }
 
 uint public x = 100;
 
 mapping(uint => Camper) public campers;
 
 function setHappy(uint index) public {
   campers[index].isHappy = true;
 }
 
 function surpriseTwo() public {
   Camper storage c;
   c.isHappy = false;
 }

We were warned that “c” is an “uninitialized storage pointer” (meaning, it points to slot 0). Try “surpriseTwo()”. Then, look at “x().” What happened to poor Mr. X? “x” isn’t 100 anymore. It’s 0, because “false” cast as a uint is 0. And guess what? “x” just happened to reside in the first storage slot, so when the errant uninitialized storage pointer decided to write something down it scribbled on top of “x”. Sorry, Mr. X. Unlucky for you.

Slot 0 could even be the “.length” of a dynamic array or the owner of the contract. Nothing good can come out of accidentally overwriting important data. And, it’s all important because we believe in minimalist design.

Why are Storage Pointers even useful?

There are arguments in favor of storage pointers. For one thing, they reduce the amount of data that gets passed around and that saves gas. They can also lead to readable code. Consider this usage:

function slightOfHand(uint index) public {
   Camper storage c = campers[index];
   c.isHappy = false;
}

That updates the storage and means you don’t need to repeat the more verbose form, “campers[index]”, but coders and reviewers need to be aware that “c” is really the same as writing the more verbose expression. In other words, it’s short for:

campers[index].isHappy = false;

My discomfort with this is that this is very easy to overlook, especially for learners. It contributes to mental overhead when we should be aiming for simplicity. It can lead to a surprise. Dragons!

Safety Habits

Storage pointers are advantageous when you know what you’re doing. But how can newbies operate safely while they learn? Here are some tips. Please comment!

You can generally avoid trouble by adopting these habits:

1. Never declare a mapping inside a function. There be dragons.
2. Never declare persistent storage inside a function, even if the compiler seems to let you.
3. In your functions, always explicitly declare structs and arrays as either transient “memory” or “storage” pointers. This habit helps you pause and think about what’s going on.
4. Never ignore warnings about storage pointers. This bears mentioning because they don’t stand out against other warnings that can be safely ignored. Storage pointer warnings == dragons in your vicinity. Don’t shrug it off.
5. Feel free to instantiate storage pointer aliases for verbose syntax, but form the habit of using them on a read-only basis. For example:

Product storage p = productStructs[productIdList[productListRow]];
return (p.id, p.desc, p.price);

6. Use extreme caution when setting a variable that was initialized with a reference to something in storage.

p.price = // Stop. Caution. Do you know where this is going?

Storage pointers are a subtlety in Solidity. It takes practice to learn what they are and how they work.

If this post helps you remember storage pointers exist so you’re aware that they might be in play (even unintentionally) then hopefully that will prompt an impulse to double-check assumptions, thoroughly test and reinforce awareness of the need for thorough code audits before releasing code in production.

At B9lab, we are dedicated to guiding students to the top of this field. Our hands-on training programs mentored by instructors like myself and workshops prepare students for stringent certification exams.

Start with your Ethereum Developer Certification and consider branching out to Quorum Specialist (think “enterprise”) or Quality Assurance Specialist. Maybe even check out Hyperledger, Corda and EOS. They too are interesting and different.