The problem isn't whether you can understand the code or not. The problem is whether or not there are undiscovered exploits in the code that would allow attackers to take advantage of Signal's relatively open permissions.
It's very hard to write bug-proof code. Restricted permissions would be a sensible countermeasure since Signal is a likely target.
I was down voted heavily for suggesting that! I don't really think that breaking out of its sandbox is the problem though, its touching and manipulating data from the network. All the data I worry about losing is inside the signal app.
It would be better if features could be disabled. Sadly the only way to achieve that is to fork the code base and run your own dev build.
It's very hard to write bug-proof code. Restricted permissions would be a sensible countermeasure since Signal is a likely target.