Best practice

our operations management utilises what we deem best practice. that is, we either have implemented or are striving to achieve the following in everything we implement and manage:
provisioning of new (or re-purposing of existing) infrastructure is either automated or documented transparently.
bootstrapping and configuration of infrastructure is automated transparently.
routine maintenance of infrastructure is automated transparently.
extraordinary maintenance of infrastructure is automated or documented transparently.
output/logs from infrastructure are aggregated and available transparently for analysis.
issues with infrastructure are transparently visible, obvious and trigger alerting.

Technologies and platforms

to realise these objectives, some of the technologies and tools we utilise include:
ansible for instance configuration and maintenance.
cloud-config for instance bootstrapping.
custom orchestration: rubberneck utilises cloud-config like maintenance configurations.
docker for applications with complex dependencies that benefit from being shared and run by an expanded audience.
github actions with self-hosted runners for ci build, test, release, deployment.
letsencrypt/certbot for automated cert acqisition and renewal.
nginx for cert serving, reverse proxying, load balancing.
mongodb, pgsql for application state.
prometheus, promtool. loki, grafana for log aggregation and dashboarding.
route53, cloudflare for dns management, load balancing, health/uptime monitoring.
ssh, gpg, wireguard and ed25519/curve25519 are the underpinnings of our security infrastructure.
terraform for instance provisioning and bootstrapping.

Requirements

a successful candidate would likely have experience, skills and aptitudes that include:
given our mission, a strong, personal motivation and conviction to contribute in the transaction privacy space is an essential aspect.
written and spoken english proficiency or the ability to interact with written and spoken english effortlessly using technology.
championing technology, tools and practices that facilitate the mission. understanding when, why and how to use and champion what.
an ability to rapidly prototype and demonstrate the benefits and pitfalls of solution proposals. any aptitude for dashboarding tools like react, vue, grafana or others is useful.
networking
a practical understanding of networks, dns, tls, firewalls, port forwarding, reverse proxying, traffic shaping and packet dropping
distributed source control
git and github/gitlab and the use of their associated tooling and configuration should be second nature. a github/gitlab profile that demonstrates a history of best practice is always impressive. if privacy or other concerns prevent this, then some other way of conveying this experience is helpful.
a bias toward transparency is essential.
our intention is to make everything we do discoverable, understandable and repeatable by others. ie:
see my patch implementation at this url: …
is infinitely preferable to:
i fixed all the things and they work great now.
an appreciation of security concerns including (an aptitude to learn quickly is fine):
what aspects of an implementation or configuration should be deemed secrets?
what aspects of an implementation or configuration can benefit from being transparent/public and why?
a practical understanding of gpg, ssh, wireguard, rsa, ed25519 and curve25519 and when to use which is essential.
being adept at spotting vulnerabilities and weaknesses.
being able to read bash will make it easier to understand what’s going on. we use a lot of it in both documentation and implementation.
understanding when to containerise or componentise a thing and being able to articulate the justifications for either.
understanding of checksums and hashing.
understanding what type of updates should happen automatically and what type should be more considered or deliberate.
weekly status report (more frequently when there are critical issues in the pipeline) including:
a written summary (internal, notion) of what was achieved, what is planned and any blockers.
two zoom meetings (immediate infrastructure team, wider engineering department), articulating the same, with webcam on.
out-of-hours reachability and availability for mission-critical or emergency issue resolutions.
being relaxed about admitting mistakes or problems with a view to sharing learnings and improving the overall sittuation.

Good to Have

some skillsets and other considerations that will set you apart, include:
being available during pst, cst or est hours is a bonus as eet and aest are currently covered.
being willing and able to hold peers accountable for the responsibilities above.
knowing how to use cryptocurrencies, create and manage accounts/keys/wallets, instigate and analyse transactions and other extrinsics or smart contracts.
experience running a complete/full blockchain node (for any chain but, substrate or ethereum are especially useful, understanding tokenomics, validation or staking dynamics, more so).
understanding distributed and decentralised systems, incentivisation, ipfs, tor, etc…
rust, go, javascript and python (or any async capable language) skills.
cryptography primitives and/or blockchain at the protocol or white-paper level.