FULL STEAK DEVby VICTOR NAKORIAKOV
🌿 May 2022

How Sanctions Affect a Developer in Russia

On the 24’th of February 2022, Russia started a war against Ukraine. The global society quickly began to react by putting massive economic sanctions on Russian politics, officials, oligarchs, goods, and whole economic sectors. Regardless of how precise these sanctions were, they inevitably hit regular Russian citizens and IT specialists in particular. In a few weeks we were able to become the most sanctioned country after all.

In this post, I’d like to describe what has changed in a few weeks from the perspective of a humble developer and CTO living in Moscow. In no way is this post an attempt to explain how to work around limitations and try to live as it was before the war. Instead, this post showcases what can happen when service providers refuse to give a service in a blink of an eye.

I clearly understand that the discomfort we’ve got is nowhere near the suffering of any Unkrainian. I feel shame and mental pain for all those dotards from Kremlin and insane RuZZians supporting the war. I’ve read sentiments that banning Russians from services is not a correct business behavior. They paid the contracts in advance, and they were not in charge of the invasion and killing of people. I disagree. I’m Russian. The limitations, restrictions, and bans are the very least we deserve for this war.

I stand with Ukraine 💙💛

No Visa/Mastercard means no SaaS

In the first days of March, Visa and Mastercard stated they would stop issuing and processing all Russian bank debit/credit cards within a week. Before the first batch of sanctions, there were rumors that Russian banks would be cut off from the wire transfer system called SWIFT. It sounded very worrying because the business would have a hard time paying for goods they import. Many didn’t believe in such a scenario until the very last moment. But the reality outperformed even the negative expectations. Now we have neither SWIFT in major banks nor Visa/Mastercard.

Sure, the existing cards still work for inter-country payments because banks intercept such operations and perform them over the national payment system. But the charges for international services are no longer possible. For our company, it means termination or severe degradation of:

  • Google Workspace (E-mail and Drive)
  • Google Cloud Services (Hosting, Kubernetes, Storage, App platform)
  • Linode (Hosting)
  • Digital Ocean (App platform, Storage)
  • GitHub (Code hosting, Collaboration)
  • CircleCI (Continuous Integration)
  • Sendgrid (Email gateway)
  • MailChimp (Newsletters)
  • Odoo (ERP system)
  • Microsoft 365 (Office Apps)
  • Adobe Suite (Creative Apps)
  • a dozen less critical systems and tools

Thinking about this stop-list back then was thrilling enough to start a panic. If all these systems were shut down in a moment, the IT backbone of the business we have been building for the last decade would go to pieces that are nearly impossible to glue back. Luckily we were given some time to move, escape, backup, and migrate.

Currently, we are balancing a tricky situation with the help of friends and family members from abroad. They kindly agreed to pay for cheaper services. This, of course, cannot last forever.

Google no longer accepts payments directly, but their local reseller partners can still work with existing customers using inter-country invoicing. Such big luck we were able to become an “existing” customer the last day it was possible. Google offers a wide variety of services that became a part of daily workflow years ago. Not all services can be easily replaced with an offering from another platform, though: Firebase, Google Spreadsheets, and DataStudio, to name a few. It would be painful and take weeks to months to recover if we were forced to leave the platform entirely.

Another money-hungry SaaS is Odoo ERP. Even though it is an open-source system, we use its hosted variation with proprietary enterprise add-ons we rely on. The service requires €1K per month. This is a hard nut because such an amount is very noticeable for individuals helping abroad, both in receiving and spending.

Other than most critical systems, we had to cancel those we love and use daily. Bye-bye, Linode with your great support, cozy Digital Ocean, agile Trello, the canonical MS Office, and many others. I will miss you.

We can operate now, but the land is very shaky. Every day I’m waiting for new obstacles in the “you are not allowed to use this” series, and every week I get a new one. In summary, for the last couple of months, we as developers do not develop products, the business, or the good. The only thing we do is try to dodge and recover.

Free-will cancellation

Even if you find a way to go on with payments, the second wall waits for you. Many companies have blocked accounts that belong to Russian users as a protest against the invasion. In our case, the companies are:

  • Odoo (Belgium)
  • CircleCI (United States)
  • MailChimp (United States)
  • Grammarly (Ukraine)
  • Spotify (Sweden)

Yesterday I got a message from Odoo telling me they no longer offer ERP as a service to Russian companies. Doh. Odoo ERP is the central part of our operations with tens of gigabytes of data about purchases, payments, sales, manufacturing ops, customers, goods movement, everything… Almost all employees use it every day to act as one organism. I want to thank Odoo at least for giving us some time to move away.

CircleCI has something to say too. Consider making an innocent routine PR and seeing that the CI check fails and you cannot merge. Why is it failing? You open the log and see it is failing not because of syntax error, a failed test, or a request timeout. It fails because you’re Russian! Now, the day’s challenge is to figure out how to fix this bug.

MailChimp is on the same boat. If you’re in Russia, you can no longer use their service. It might not be a significant loss. You can quickly move the list of e-mails to one of many other similar services unless you rely on the service-specific functions. Transaction-mail triggers, purchase history, and product catalog integration are hard to migrate because of the lack of some widely adopted standards. To be honest, MailChimp is the last service I miss right now. Since Feb 24 I see no way our company can send 10000 marketing e-mails entitled with “Hey-hey, fun days, do not miss an opportunity to buy the great gadget this weekend”. However, I’m sure there’re plenty of businesses where e-mail marketing is the core part of operations.

Grammarly and Spotify are not developer tools, but I listed them here because they were an integral part of my work experience. Grammarly is a fantastic product that creates an illusion my colleagues and I know English (in docs, comments, letters, and articles). Spotify was an ideal noise isolator in the open space environment. Now they have gone too, sad.

I know JetBrains also blocked Russian users, although the company used to have many employees and relations in Russia. We are not their users because of the tech stack, but I can imagine how it hurts PHP, Ruby, and JVM devs.

Network fragmentation

One specific example of network problems is our national delivery service CDEK. Very roughly, CDEK is FedEx or DHL of Russia. Two days in April, they experienced a massive DDoS attack. What did they do? They have dropped a curtain wall for any API requests outside of Russia. Nice! We can no longer dispatch our orders because our ERP system is in the EU. OK, we’ve dug a proxy tunnel from the EU to Russia using a local cloud provider just to make the API requests seem legitimate. This is just one example, and we had to dodge much more during the last weeks.

The split between “your Internet” and “their Internet” becomes more and more apparent. No, we’re not isolated (yet). Still, we experience a cascade violation of services due to the brave fight of our Roskomnadzor (acronym of Russia Communication Control) against fake news and bad foreign actors. They also installed Deep Packet Inspection hardware on the key points of the traffic for non-trivial blockings 💪 But, we have VPN, and we have Tor. Fuck you, Roskomnadzor. And your DPI.

As a user, you have to enable VPN to read some dev docs and disable to enter a bank app. Then, enable to read some real news, disable to manage the local domains DNS… You’ve got the idea.

It’s more an annoying part if you’re a developer. Various hostings and CDNs become more and more suspicious if a user from Russia accesses some international site and vice versa. This often results in user experience degradation by causing synthetic delays, slow checks, and presenting CAPTCHAs. Furthermore, you should have your fingers crossed if you host on a European server because its IP can easily start to belong to a blacklisted address pool.

Protestware

Besides sanctions from companies, individuals also express their position. For example, an author of the node-ipc NPM package has included some malicious code that replaces file contents with heart emoji if the code is run on a server with Russian IP address.

Although such a way to express the anti-war position has met much criticism from the open-source community, the case itself shows what’s possible. The node-ipc package is a transitive dependency of some large and popular frameworks (Vue, for example), potentially affecting many users. Consider a similar protest but with a more “interesting” backdoor, such as stealing data or destroying servers at day X. To make things even trickier, no one forces its author to show the changes on GitHub; uploading a compromised version directly to NPM is enough.

Mobile dev

A little cherry on the Russian Android dev’s cupcake: now, you can’t. So, you have a crazy idea for the next big music app, an interested investor, and a mentor? It does not matter. You can’t upload the app, and you can’t receive the payments. Your users can’t update the app (new users can’t install it, yay), nor can they pay. In short, you can’t touch the Android business.

I don’t know anything about the iOS business now, but I guess it goes in the same direction.

Many companies were built around their successful mobile apps and games. Now they are in trouble. The only viable solution is to re-locate.

Lessons learned

Unless you’re a very specific and isolated business, it’s almost impossible to build your IT entirely immune to external bans and restrictions. I wish you (and we all, as humanity) will not face another catastrophe leading to the service denial of the same scale. However, I’ve read plenty of publications showing us that some targeted “sanctions” are possible. Would you be surprised if you read a blog post about AWS/GCP/Azure blocking someone without a chance to resolve the issue, causing them to run out of business? I won’t be surprised.

The main source of problems is too much centralization and vendor locks. Here are some ways to minimize the risks.

Every time you create an account anywhere, think twice

What will your business do once the service provider disappears in a moment or ban you without a chance for recovery? It is OK to go on with ad-hoc tools or if the potential damage is all about discomfort and not about business process rebuild.

Prefer SaaS with an open-source model

When you have to rely on an external service provider, try to find a variant where you can, technically, deploy your own local copy of the service, transfer all your data, and become independent of the original service provider.

Enjoy the comfort of reliable hosting, auto-scaling, and auto-updates as long as everything is fine, but have a way to move quickly in the case of an emergency.

Good examples of such products are Discourse, GitLab, Plausible, and various database-as-a-service providers.

There’re partially open services with the core functionality available and bells’n’whistles locked. This alternative is worse, but viable enough if you can agree to lose the extras someday, in theory. Odoo and GitHub follow such a model, for example.

Stay away from completely proprietary solutions

Even if you can take your data away, what would you do with it when the service is stopped? If you build your business around a thing you cannot manage, can it grow and stay healthy for ten years? 20? 30? You’re too vulnerable to the decisions of a small group of people, and people change their minds.

SAP, MS Dynamics? No, thank you. Google Firebase? Nope. Shopify on the scale? No! App Store and Google Play? Huh, no choice; we need more independent stores and distribution channels, though.

In many cases, an open-source alternative does exist. I’d prefer a free alternative with inferior functionality over a vendor-locked solution with no access. If you insure, ask yourself again: “What will your business do once the service provider disappears in a moment?”.

Prefer standard protocols and interfaces

It’s OK to employ a proprietary SaaS if all you need from it is to serve a well-known protocol. For example, if you need to send e-mails, it is OK to go ahead with SendGrid because it offers the vendor-neutral SMTP gateway. In case of failure, you can easily switch to another proprietary provider offering SMTP. Just don’t rely on the proprietary API where possible.

The same for things like S3 storage, managed Kubernetes cluster, Docker container registry, and so on. It is OK to use any cloud platform as long as you don’t get tied by their intrinsic features and APIs.

Take care of dependencies

Pin versions; use lock files. Do not add a dependency if all you need is a 20-line function you can audit and vendor to your codebase. Read changelogs when you update dependencies; read Hackernews to know the threats.

Final thoughts

We’ve done some workarounds and arranged kludges here and there. Does it form a solid basis for a sustainable future? Obviously, no. I’ve heard many people here really think Russian IT devs will stand up, show everyone the fist, the will, the grin, and re-build everything from scratch. So naive.

China and Iran are the examples we can think about while making some future projections. Yes, there are some projects originating there and known worldwide, but a thousand times more came from free countries.

I think we’ve lost our chance for a bright future in Russia. We were too lazy and blind to recognize the coming tragedy and throw out people in charge early. Now we see the result.

Possibly the best thing one can do is leave Russia to stop paying taxes here, lower consumption (to lower others’ profits which lowers taxes), and slow down the economy due to the lack of brain contribution. No money, no guns ☮️️ Pray for the war to stop. Glory to Ukraine.