Operating System Choices (Windows/Mac/Linux)

Choosing the right operating system might seem significant. Primarily, it's a preference for what you like and know. Still, there is the fact that most data platforms that run on a server will run on a Linux-based OS system. Working on Linux OS on the client might give you skills you can reuse, but you can also have that with Windows with WSL¹ and MacOS running a Darwin-based Linux.

Your employer also defines it. If you are a Microsoft shop, you use tools such as Power BI, Visual Studio (not Visual Studio Code), and C#. This requires using Windows or at least a VM with Windows.

If you work at a startup and need great hardware that is easy to use, the company will probably provide you with the latest MacBook with MacOS installed. However, if you are a power user or need your Dotfiles, you may not use anything other than a Linux-based operating system. We will look later at fundamental Linux commands that make the life of every data engineer easier.

Virtual Machine (VM)

As mentioned, you could run MacOS and Windows in a VM with VMware or Parallels. These are not native installations, but close to it, and they allow you to do most things.

The same goes if you are on Windows; instead of using WSL, which sometimes can get tricky with companies' proxies and network routing, you could use a Linux VM locally or somewhere hosted that you just SSH into or an advanced example with Nix. There are other solutions to explore; e.g., your whole machine could be a VM provided by your company or deploy a VS Code server to run VS Code instances inside your company network.

ENV variables

The next layer that you commonly use is ENV variables. This is already a little more advanced. But think of your reproducible environments with your co-workers or managing different environments (dev/staging/prod) instead of hard copying all settings, which won't work on other environments with different OS or other expectations.

If you type env in a Linux-based OS terminal, you can see all your local env sets. To illustrate some, I have set these ENVs:

Copy code
❯ env
AIRFLOW_HOME=~/.airflow
SPARK_HOME=~/Documents/spark/spark-3.5.1-bin-hadoop3.3
MINIO_ENDPOINT=http://127.0.0.1:9000
GITHUB_USER=sspaeti
AWS_SECRET_ACCESS_KEY=my-secure-key
AWS_ACCESS_KEY_ID=my-access-key

These can be set in a projects-repositories folder, usually in .env, and which will be picked up automatically. However, the recommended approach is using SSO CLI tools (like aws sso login or gcloud auth login), which will automatically populate credentials in the expected locations, or alternatively adding them to your shell config (~/.bashrc, ~/.zshrc).

Docker and Container Images

Another virtualized environment is Docker, and specifically Dockerfiles. Docker is the engine that runs your Dockerfile on all platforms and architectures, letting you create a container image and build it for Linux on a Windows machine.

That makes containers so powerful: you can package and containerize complex data engineering requirements into a single Dockerfile, and everyone can run it on any machine—whether locally, in CI/CD pipelines, or orchestrated in Kubernetes clusters. Think of container packages on ships that transport goods; the breakthrough was the standardized container size that fits on every boat; every harbor could maneuver them. Similarly, container images have become the standard for packaging data and software ecosystems, with formats originally defined by Docker now being widely supported across different container runtimes and platforms.

A simple nginx (webserver) example:

Copy code
# Use the official NGINX image from Docker Hub
FROM nginx:latest

# Copy your custom NGINX configuration file (if you have one)
COPY nginx.conf /etc/nginx/nginx.conf

# Copy static website files to the appropriate directory
COPY . /usr/share/nginx/html

# Expose the port NGINX listens on
EXPOSE 80

Docker also supports different instructions that you can use in a Dockerfile.

Opening and Editing a File with Nano/Vim

Editing or creating a new file might not be as easy as it seems. Command line text editors such as Nano or Vim can be used for this task. Recommended is Nano, which displays the shortcuts to save or exit. Vim can be intimidating at first, but it's a worthwhile investment when working 8 hours a day on the terminal, even more so Vim Motions.

Example of editing above Dockerfile in Nano.

Basic Linux Tools and Commands

In addition to the Linux basic commands you have probably used or encountered like cp, mv, ssh as seen below, which are also super helpful on a server, we focus on the data engineering Linux commands you run on your laptop, where you can install things.

Image from Linux is a MUST. Seriously...| Also, check more on the book Efficient Linux at the Command Line by Daniel J. Barrett.

Most tools are Python-related to achieve the core tasks of a data engineer: ingestion of data, transforming and serving it to the organization or users. But the additional DE Linux commands I often use to quickly check an API, copy something over, or check processes are:

• curl: Quickly check an API is available through the cmd line.
• make / cron: Simple orchestration with the command line. More on this in the next chapter
• ssh / rsync: Ssh to connect to another machine and Rsync for a fast, versatile, synchronization tool to quickly back up or move data from your machine to the server.
• bat: Show data of a file nicely format and git integration.
• tail: Displays the last part of a file, which is helpful if the file is big and cat/bat would take long.
• which: Locate a program in the user's path to check if the right tool is running.
• brew: MacOS-specific package manager is the easiest way to install tools and cmd line utils.

Related to the above basic Linux commands:

• grep: Used for everything attached to an existing run. E.g. quickly search AWS env variables:

Copy code
❯ env | grep AWS
AWS_ACCESS_KEY_ID=my-access-key
AWS_BUCKET=my-bucket
AWS_SECRET_ACCESS_KEY=my-secret

• ps aux and htop: To check the current process. Ps is also handy in combination with grep (ps aux | my-program.py)

• rg and fzf: Ripgrep (rg) is a recursive line-oriented search tool that searches through all files, and fzf is a fuzzy finder. In combination, you can interactively search fuzzy find the content of Python files in the current folder easily with rg -t python "def main" . | fzf. (Also check out Recursive Search in Terminal with fzf, this will change your cmd-line life with reverse search ctrl+r).

Simple Orchestration

The core responsibility of a data engineer is to orchestrate different jobs in the correct order and fully automate them. We use data orchestrators (Airflow, Dagster, Prefect etc.), but Linux also covers us.

Makefile and cron jobs are out of the box and installed on every Linux system. For example, Makefiles let us store a combination of commands like this:

Copy code
API_URL := "https://api.coincap.io/v2/assets"
DATA_DIR := /tmp/data

etl: extract transform load

extract:
  mkdir -p $(DATA_DIR)
  curl -s $(API_URL) | \
    jq -r '.data[] | [.symbol, .priceUsd, .marketCapUsd] | @csv' > \
    $(DATA_DIR)/crypto_raw.csv

transform:
  ./scripts/transform_data.sh
    
load:
  cat $(DATA_DIR)/crypto_raw.csv | \
    sort -t',' -k3,3nr | \
    head -n 10 > $(DATA_DIR)/top_10_crypto.csv

clean:
  rm -rf $(DATA_DIR)/*

Running make extract will create download data from the HTTPS API and store it as CSV, which we can check with tail:

Copy code
❯ make extract
mkdir -p /tmp/data
curl -s "https://api.coincap.io/v2/assets" | \
                jq -r '.data[] | [.symbol, .priceUsd, .marketCapUsd] | @csv' > \
                /tmp/data/crypto_raw.csv

❯ tail -n 3 /tmp/data/crypto_raw.csv
"ZEN","25.2499663234287359","399199442.5767759717054100"
"SUSHI","1.4507020739095067","381986878.5063751499688694"
"JST","0.0384023939139102","380183699.7477109800000000"

Combining these commands can be quick and super powerful. Make is just one example of storing and checking the commands into git so everyone can use them.

Crontabs are another way to schedule them daily, for example.

Copy code
pip freeze | grep SQL

Copy code
❯ bat /tmp/data/crypto_raw.csv | tr -d '"' | cut -d',' -f1,3 | sort -t',' -k2 -nr | head -n 4
BTC,1920648934960.3101078883559601
ETH,386675369242.2018025632681003
XRP,161734797349.4803555794799785
USDT,137222181131.1690655355161784

Copy code
# Split large CSV while keeping header
head -n1 large_file.csv > header.csv
split -l 1000000 --filter='tail -n +2' large_file.csv chunk_
# Add header back to each chunk
for f in chunk_*; do cat header.csv "$f" > "with_header_$f"; done

IDE (Working environment)

An integrated development environment (IDE) is where we program our code and get code completion, linters, and AI assistance to make us (hopefully) more productive.

Popular IDEs are with their used based on the StackOverflow Survey 2024:

• Visual Studio Code (73.6%) - Microsoft's lightweight but powerful source code editor with extensive plugin support and language coverage.
• Visual Studio (29.3%) - Microsoft's full-featured IDE, powerful for .NET development and enterprise applications.
• Other editors sorted percentage-wise are IntelliJ IDEA (26.8%), Notepad++ (23.9%), Vim (21.6%), PyCharm (15.1%), Jupyter (12.8%), Neovim (12.5%), Sublime Text (10.9%), Eclipse (9.4%), Xcode (9.3%)

Not even on the map 2024 were the IDEs that go all in with AI:

• Cursor - A VS Code-based editor explicitly built for AI-assisted development, featuring GitHub Copilot integration and specialized AI tooling for code completion and refactoring.
• Windsurf - An AI-first code editor designed to streamline development workflow with features like natural language code generation and intelligent code suggestions.
• Zed - A high-performance, multiplayer code editor with AI capabilities created by former Atom developers.

Codespaces and Workspaces

In addition to IDEs that are usually installed locally, we also have codespaces (or workspaces, depending on the naming) that live in the browser. These are super handy because everyone has the same environment, and the days of "does not work on my machine" are gone.

These tools include GitHub Codespaces, Devpod, Replit, Stackblitz, CodeSandbox Gitpod, and many others.

Notebooks

In addition to IDEs and Codespaces, you can use a notebook that runs locally or in the cloud. This option is generally more flexible and allows you to visualize results and document the code. However, putting it in production has a downside: It's harder to restart, backfill, or configure with different variables.

It’s more flexible and easier to get started, but transitioning notebooks to production remains challenging even on platforms like Databricks, which are designed to support a development-to-production workflow.

Notebooks like Jupyter Notebook / JupyterHub, Apache Zeppelin, or Databricks Notebook. Newer versions of Jupyter Notebooks with more integrated features and a robust cloud behind them are Deepnote, Hex, and Count.co, Enso, or MotherDuck, which combines the flexibility of notebooks with the power of DuckDB's analytics engine.

Git Version Control

Git is probably the most used version control in data engineering nowadays. There was a time of TortoiseSVN and others.

As a data engineer, you need to version your code and product to easily roll back in case of error or work together as a team. The most common git workflow are:

Copy code
git pull origin main # Pull latest changes
git status # Check status of your changes
git add pipeline.py #stage
git commit -m "fix: update extraction logic for new API version" #commit
git push origin main # Push to remote repository
git checkout -b feature/new-data-source # Create and switch to a new branch

For more complex operations, consider using a Git GUI client. Some popular options include GitKraken, SourceTree, Lazygit (terminal UI), and many more.

Beyond Languages

Beyond programming languages, you must get to know various databases and their concepts, such as relational database theory. It does not matter which SQL dialect you learn, as they are all related, but knowing the fundamentals of a specific database, such as Postgres, DuckDB, or a NoSQL database, will help you on your journey.

Python libraries and frameworks are the last we observe and where you can spend most of your time. Instead of learning as many as possible, I suggest investing in a few used at your company and where you benefit most.

Typical starter libraries include DuckDB (a powerful in-memory transformation library and database with scale-up capabilities via MotherDuck²), Pandas (flexible data manipulation), PyArrow (optimized for columnar data), Polars (fast and scalable DataFrame library), and PySpark (for distributed data processing with Apache Spark).

Python Libraries

There are many more libraries available, especially when you need to quickly access an API or perform a task that a CLI can't. Some key libraries can be beneficial depending on the use case you are working on.

Data Ingestion:

• Requests - HTTP library for API queries and web scraping
• BeautifulSoup - HTML parsing library for web scraping

Developer Tools:

• uv / pip - Package installers for Python, with uv being a modern, fast alternative to pip
• Ruff - Fast linter and code formatter
• Pytest - A testing framework for Python

Data Validation:

• Pydantic - Data validation for Python objects
• Pandera - Schema validation for dataframes
• Great Expectations / OpenLineage - Data quality validation framework and data lineage tracking tools

We could go on forever. Libraries exist for virtually everything: data ingestion, orchestration, BI tools, you name it. We could discuss setting up a Python project (it's not a solved problem, and there are many ways of doing it), discuss DevOps and how to use a simple Helm script, set up a local storage system that mimics S3, and more.