Alin Li NiFi Course
Data Flow Adventures — Learn how data moves like magic!
v1.0 • 4 Modules • Ages 9-14
Data Flow Adventures — Learn how data moves like magic!
v1.0 • 4 Modules • Ages 9-14Discover the amazing tool that moves data around like a super-smart delivery service!
Data is information. It's everywhere! Your name, your school grades, the temperature outside, photos on your phone, messages you send — all of these are data.
Companies and organisations collect tons of data every second. Think about a weather station recording temperature every minute, or a shop tracking what people buy. All this data needs to move from one place to another to be useful.
Imagine millions of letters and packages arriving at a post office every day. Someone has to sort them, read the addresses, decide where they go, and deliver them. Apache NiFi does exactly this — but for data!
Apache NiFi is a free, open-source software that helps you collect, move, and deliver data between different systems automatically. It was originally created by the NSA (National Security Agency) in the USA and later given to the open-source community through the Apache Software Foundation.
The name "NiFi" comes from "NiagaraFiles" — like the powerful Niagara Falls, it handles huge rivers of data flowing from one place to another!
NiFi is completely free to use, modify, and share. This aligns perfectly with the Alin Li mission of reviving the spirit of open computing. No paid licenses, no subscriptions — just powerful, free software for everyone!
NiFi is like a super-smart delivery service for data. Here's what it can do:
Grab data from files, websites, sensors, databases, and more
Change data format, filter what you need, combine different sources
Send data to databases, files, cloud services, or other apps
See exactly where your data is and what's happening to it in real-time
Encrypt data, control who can see it, and track every change
Process millions of data items per second without breaking a sweat
NiFi uses a concept called "data flows". A data flow is like a recipe that tells NiFi: "Take data from HERE, do THIS to it, then send it THERE."
Here's a simple data flow:
Raw materials (data) come in → workers on the line (processors) shape and transform them → finished products (processed data) come out at the end. NiFi is the factory manager that controls the entire line!
Learn the NiFi vocabulary by tapping each card:
Even as a young learner, understanding data flow is super valuable because:
1. What does NiFi stand for?
2. What is a FlowFile in NiFi?
3. NiFi is open-source software. What does that mean?
4. What is the Canvas in NiFi?
Let's get NiFi running on your computer — it's easier than you think!
Before installing NiFi, make sure you have:
At Alin Li, we love Linux because it's free, open-source, and perfectly suited for running server applications like NiFi. Most NiFi installations in the real world run on Linux!
NiFi needs Java to run. We'll install OpenJDK — the free, open-source version of Java.
On Ubuntu/Debian Linux:
You should see something like: openjdk version "21.0.x" — that means Java is ready!
Download OpenJDK from adoptium.net — it's the free Java from the Eclipse Foundation. Run the installer and make sure to check "Set JAVA_HOME variable" during installation.
Now let's download NiFi itself!
.zip file for all platforms).On Linux (using the terminal):
The version number (2.1.0) will change over time. Always check the official NiFi website for the latest version before downloading.
Time to launch NiFi and see the magic happen!
On Linux/macOS:
On Windows:
NiFi takes about 30-60 seconds to start up. It's loading all its powerful processors and tools in the background. Wait until you see "NiFi has started" in the log.
NiFi has a beautiful web-based interface. Open your browser and go to:
NiFi 2.x generates a random username and password on first start. You'll find them in the log:
NiFi 2.x uses HTTPS (secure connection) by default on port 8443. Your browser may show a security warning — this is normal for a local self-signed certificate. Click "Advanced" → "Proceed" to continue.
Once logged in, you'll see the NiFi Canvas — your blank workspace ready for building data flows!
The NiFi interface has these main parts:
Top bar with icons for adding processors, labels, and other components
The big white area where you drag-and-drop to build flows
Shows how many processors are running, stopped, or have errors
Controls to start, stop, and configure your data flow
1. What do you need to install BEFORE NiFi?
2. What is the default web address to access NiFi?
3. Which command starts NiFi on Linux?
Time to build your first data flow — drag, drop, connect, and watch data move!
Every NiFi data flow is made from just a few building blocks:
That's it! With just processors and connections, you can build incredibly powerful data flows. Let's learn about each piece.
Processors are the workers of NiFi. Each processor does ONE specific job. NiFi comes with over 300 built-in processors! Here are the most important ones for beginners:
| Processor | What It Does | Category |
|---|---|---|
GetFile |
Reads files from a folder on your computer | Source |
PutFile |
Writes data to a file on your computer | Destination |
GenerateFlowFile |
Creates test data for experimenting | Source |
LogAttribute |
Prints FlowFile details to the log (for debugging) | Utility |
UpdateAttribute |
Changes or adds labels (attributes) to a FlowFile | Transform |
ReplaceText |
Finds and replaces text inside FlowFile content | Transform |
RouteOnAttribute |
Sends data to different paths based on conditions | Transform |
ListenHTTP |
Receives data sent over HTTP (from websites, IoT, etc.) | Source |
InvokeHTTP |
Calls websites or APIs to get or send data | Utility |
Let's build the simplest possible flow to see NiFi in action!
GenerateFlowFile and add it.LogAttribute and add it.5 sec (so it generates data every 5 seconds).You've just built your first NiFi data flow! Check the NiFi logs to see the FlowFile attributes being printed. You can find logs at: logs/nifi-app.log
Let's build a flow that automatically moves files from one folder to another!
~/nifi-input and ~/nifi-outputGetFile processor → configure "Input Directory" to /home/yourname/nifi-inputPutFile processor → configure "Directory" to /home/yourname/nifi-outputWithin seconds, the test file will magically appear in the output folder! 🪄
The lines between processors are called Connections. Each connection has a queue where FlowFiles wait.
Imagine a conveyor belt between two workers. If Worker B is slower than Worker A, packages pile up on the belt. In NiFi, FlowFiles pile up in the queue. You can see exactly how many are waiting by looking at the number on the connection!
Queue features you should know:
Drag each processor name to the correct description:
1. What is the correct order of a basic data flow?
2. What does the GenerateFlowFile processor do?
3. What happens when a NiFi queue gets full?
Apply your NiFi skills to exciting real projects — IoT sensors, web APIs, and file routing!
Collect temperature data from an ESP32 sensor and save it with NiFi!
The Idea: Your ESP32 reads a temperature sensor and sends the reading to NiFi over WiFi. NiFi receives it, adds a timestamp, and saves it to a file. This is exactly how real IoT systems work!
This sketch connects your ESP32 to WiFi and sends temperature to NiFi every 10 seconds:
ListenHTTP → set Port to 9090 and Base Path to /iotUpdateAttribute → add property: filename = temp_${now():format('yyyyMMdd_HHmmss')}.jsonPutFile → set Directory to /home/yourname/iot-dataEvery 10 seconds, a new JSON file will appear in your iot-data folder with the temperature reading from the ESP32! You're now running a real IoT data pipeline!
Fetch weather data from a free API and save it automatically!
The Idea: NiFi will call a free weather API every 5 minutes, get the current weather data, and save it to a file. Over time, you'll build a weather history database!
GenerateFlowFile → Scheduling: Run every 5 minInvokeHTTP → set "HTTP URL" to a free weather API like:https://api.open-meteo.com/v1/forecast?latitude=25.29&longitude;=51.53¤t;_weather=trueUpdateAttribute → set filename: weather_${now():format('yyyyMMdd_HHmm')}.jsonPutFile → Directory: /home/yourname/weather-dataOpen-Meteo is a completely free, open-source weather API — no API key needed! It's perfect for learning and aligns with our open-source values.
Automatically sort files into folders based on their type!
The Idea: Drop any file into an "inbox" folder, and NiFi will automatically sort it — PDFs go to one folder, images to another, text files to a third!
inbox, sorted/pdf, sorted/images, sorted/textGetFile → Input Directory: your inbox folderRouteOnAttribute → Add routing rules:
${filename:endsWith('.pdf')}
${filename:endsWith('.jpg'):or(${filename:endsWith('.png')})}
${filename:endsWith('.txt')}
PutFile processors — one for each sorted folderThe ${filename:endsWith('.pdf')} syntax is called NiFi Expression Language (EL). It's a powerful mini-language for working with FlowFile attributes. You'll use it a lot as you build more complex flows!
You've learned the fundamentals of Apache NiFi! Here are some exciting things to try next:
Use PutDatabaseRecord to store data in MySQL, PostgreSQL, or SQLite
Use PutEmail to send yourself alerts when something interesting happens
Get data from one API, transform it, and send it to another
Connect NiFi to Grafana to visualise your IoT data in real-time
1. Which processor receives HTTP requests from an ESP32?
2. Which processor sends HTTP requests to an API?
3. What does RouteOnAttribute do?
4. In NiFi Expression Language, what does ${filename:endsWith('.pdf')} check?
Congratulations on completing the Alin Li NiFi Course — Data Flow Adventures!
You've learned how to install NiFi, build data flows, and create real IoT and API projects. Keep exploring, keep building, and remember:
"Reviving the spirit of open computing" 🌟