Mõista andmete liikumist

Moodul 1

🌟 What is Apache NiFi & Why It Matters?

Discover the amazing tool that moves data around like a super-smart delivery service!

📦 First — What is Data?

Data is information. See on kõikjal! Sinu nimi, koolihinded, välistemperatuur, telefonis olevad fotod ja saadetud sõnumid — kõik need on data.

Companies and organisations collect tons of data every second. Think about a weather station recording temperature every minute, or a shop tracking what people buy. All this data needs to move from one place to another to be useful.

💡 Think of it like a Post Office!

Imagine millions of letters and packages arriving at a post office every day. Someone has to sort them, read the addresses, decide where they go, and deliver them. Apache NiFi does exactly this — but for data!

🐝 Tutvu Apache NiFiga!

Apache NiFi is a free, open-source software that helps you collect, move, and deliver data between different systems automatically. It was originally created by the NSA (riiklik julgeolekuagentuur) USA-s ja anti hiljem avatud lähtekoodi kogukonnale Apache Software Foundation.

The name "NiFi" comes from "NiagaraFiles" — like the powerful Niagara Falls, it handles huge rivers of data flowing from one place to another!

🔓 See on avatud lähtekoodiga!

NiFi kasutamine, muutmine ja jagamine on täiesti tasuta. See sobib Alin Li eesmärgiga taaselustada avatud arvutuse vaimu. Tasuliste litsentside ja tellimusteta — lihtsalt võimas tasuta tarkvara kõigile!

🎯 What Can NiFi Do?

NiFi on nagu super-smart delivery service for data. See suudab teha järgmist:

📥

Collect Data

Grab data from files, websites, sensors, databases, and more

🔄

Transform Data

Change data format, filter what you need, combine different sources

📤

Deliver Data

Send data to databases, files, cloud services, or other apps

👁️

Jälgi kõike

See exactly where your data is and what's happening to it in real-time

🔒

Keep It Safe

Encrypt data, control who can see it, and track every change

⚡

Handle Huge Loads

Process millions of data items per second without breaking a sweat

⚙️ How Does NiFi Work? (The Big Picture)

NiFi kasutab mõistet „andmevood“. Andmevoog on nagu retsept, mis ütleb NiFi-le: "Võta andmed SIIT, tee nendega SEDA ja saada need SIIS SINNA."

Here's a simple data flow:

📁 Allikas
Get the data

➜

⚙️ Process
Change it

➜

📋 Järjekord
Wait in line

➜

💾 Sihtkoht
Deliver it!

💡 Think of it like a Factory Assembly Line!

Raw materials (data) come in → workers on the line (processors) shape and transform them → finished products (processed data) come out at the end. NiFi is the factory manager that controls the entire line!

📖 Key Terms — Tap to Flip!

Learn the NiFi vocabulary by tapping each card:

📄

FlowFile
tap to flip

A FlowFile is a single piece of data moving through NiFi. It has content (the actual data) and attributes (labels describing it).

⚙️

Processor
tap to flip

A Processor is a worker that does something with data — reads a file, changes format, sends to a database, etc. NiFi has 300+ processors!

🔗

Connection
tap to flip

A Connection is the pipe between two processors. It holds a queue where FlowFiles wait before being processed.

🖥️

Canvas
tap to flip

The Canvas is the main workspace where you drag-and-drop processors and connections to build your data flow visually!

📊

Provenance
tap to flip

Provenance is NiFi's history book — it tracks every single thing that happened to every FlowFile. Perfect for debugging!

📂

Process Group
tap to flip

A Process Group is like a folder — it organises related processors together to keep your canvas clean and tidy.

🤔 Why Should YOU Care About NiFi?

Even as a young learner, understanding data flow is super valuable because:

IoT Projects: If you have an ESP32 or Arduino collecting sensor data, NiFi can receive and store it automatically!
Future-proof skills: Data engineering is one of the most in-demand jobs in the world right now.
Visual programming: NiFi kasutab lohistamist — alustamiseks pole keerulist koodi vaja!
It's free and open source: Ideaalne õppimiseks ühtki riaali kulutamata!

🧪 Mooduli 1 viktoriin — kontrolli oma teadmisi!

1. Mida tähendab NiFi?

A) Võrguliidese failid

B) NiagaraFiles

C) New Internet Flow Integration

2. Mis on NiFi FlowFile?

A) Protsessori tüüp

B) The main NiFi application

C) A single piece of data moving through NiFi

3. NiFi on avatud lähtekoodiga tarkvara. Mida see tähendab?

A) Vabalt kasutatav, muudetav ja jagatav

B) Only free for students

C) Requires a paid license

4. Mis on NiFi lõuend Canvas?

A) Andmebaas, kuhu andmed salvestatakse

B) The visual workspace where you build data flows

C) A type of connection

Moodul 2

🛠️ Paigaldamine ja seadistamine

Paneme NiFi sinu arvutis tööle — see on lihtsam, kui arvad!

📋 What You Need Before Starting

Before installing NiFi, make sure you have:

Arvuti running Linux (recommended!), Windows, or macOS.
Java 21 or newer installed. NiFi runs on Java, so this is essential.
At least 4 GB of RAM (sujuvaks tööks on 8 GB parem).
Veebibrauser — Firefox (our open-source favourite!) or Chromium.

🐧 Why Linux?

At Alin Li, we love Linux because it's free, open-source, and perfectly suited for running server applications like NiFi. Most NiFi installations in the real world run on Linux!

☕ Step 1: Install Java (OpenJDK)

NiFi vajab töötamiseks Javat. Paigaldame OpenJDK — the free, open-source version of Java.

Ubuntu/Debian Linuxis:

Terminal
# Uuenda oma süsteemi pakettesudoaptupdate# Paigalda OpenJDK 21sudoaptinstall openjdk-21-jdk

# Kontrolli paigaldustjava-version

You should see something like: openjdk version "21.0.x" — that means Java is ready!

🪟 Kas kasutad Windowsi?

Download OpenJDK from adoptium.net — it's the free Java from the Eclipse Foundation. Run the installer and make sure to check "Set JAVA_HOME variable" during installation.

📥 Step 2: Download Apache NiFi

Laadime nüüd NiFi enda alla!

Go to the official site: nifi.apache.org
Click on „Allalaadimised“ in the navigation bar.
Download the Binary file (the .zip file for all platforms).
Extract the downloaded file to a folder you'll remember.

Linuxis (terminali abil):

Terminal
# Liigu oma kodukataloogicd ~

# Laadi NiFi alla (uusima versiooni leiad saidilt nifi.apache.org)wgethttps://downloads.apache.org/nifi/2.1.0/nifi-2.1.0-bin.zip# Paki arhiiv lahtiunzip nifi-2.1.0-bin.zip

# Mine NiFi kaustacd nifi-2.1.0

⚠️ Version Numbers Change!

The version number (2.1.0) will change over time. Always check the official NiFi website for the latest version before downloading.

🚀 Step 3: Start NiFi!

Time to launch NiFi and see the magic happen!

Linuxis/macOS-is:

Terminal
# Käivita NiFi taustal./bin/nifi.shstart# Kontrolli olekut./bin/nifi.shstatus# NiFi hilisemaks peatamiseks:./bin/nifi.shstop

Windowsis:

Command Prompt
REM Navigate to the NiFi bin foldercd C:\nifi-2.1.0\bin

REM Start NiFirun-nifi.bat

⏳ Be Patient!

NiFi käivitub umbes 30-60 sekundiga. Taustal laaditakse võimsaid protsessoreid ja tööriistu. Oota, kuni logisse ilmub „NiFi has started”.

🌐 Step 4: Open the NiFi Web Interface

NiFil on kaunis veebiliides. Ava brauser ja mine aadressile:

https://localhost:8443/nifi

NiFi 2.x loob esmakäivitusel juhusliku kasutajanime ja parooli. Leiad need logist:

Terminal
# Leia loodud sisselogimisandmedgrep-i"Generated Username" logs/nifi-app.log
grep-i"Generated Password" logs/nifi-app.log
    

🔒 Vaikimisi HTTPS

NiFi 2.x kasutab vaikimisi HTTPS-i (turvalist ühendust) pordil 8443. Brauser võib näidata turvahoiatust — kohaliku iseallkirjastatud sertifikaadi puhul on see tavaline. Klõpsa „Täpsem” → „Jätka”.

Pärast sisselogimist näed NiFi Canvas — your blank workspace ready for building data flows!

🗺️ Quick Tour of the NiFi Interface

The NiFi interface has these main parts:

📐

Toolbar

Top bar with icons for adding processors, labels, and other components

🎨

Canvas

The big white area where you drag-and-drop to build flows

📊

Status Bar

Shows how many processors are running, stopped, or have errors

⚙️

Juhtpaneel

Controls to start, stop, and configure your data flow

🧪 Mooduli 2 viktoriin — paigalduse kontroll!

1. Mida tuleb paigaldada ENNE NiFi-t?

A) Python

B) Node.js

C) Java (OpenJDK)

2. Milline on NiFi vaikimisi veebiaadress?

A) http://localhost:8080

B) https://localhost:8443/nifi

C) http://nifi.local:3000

3. Milline käsk käivitab NiFi Linuxis?

A) ./bin/nifi.sh start

B) nifi --run

C) sudo start nifi

Moodul 3

🔗 Andmevoogude loomine

Time to build your first data flow — drag, drop, connect, and watch data move!

🧱 The Building Blocks of a Data Flow

Every NiFi data flow is made from just a few building blocks:

📥 Allikas
Processor

➜

📋 Ühendus
(Järjekord)

➜

🔄 Teisenda
Processor

➜

📋 Ühendus
(Järjekord)

➜

📤 Sihtkoht
Processor

That's it! With just processors and connections, you can build incredibly powerful data flows. Let's learn about each piece.

⚙️ Understanding Processors

Processors are the workers of NiFi. Each processor does ONE specific job. NiFi comes with over 300 sisseehitatud protsessorit! Siin on algajatele kõige olulisemad:

Processor	What It Does	Category
`GetFile`	Reads files from a folder on your computer	Source
`PutFile`	Writes data to a file on your computer	Destination
`GenerateFlowFile`	Creates test data for experimenting	Source
`LogAttribute`	Prints FlowFile details to the log (for debugging)	Utility
`UpdateAttribute`	Changes or adds labels (attributes) to a FlowFile	Transform
`ReplaceText`	Finds and replaces text inside FlowFile content	Transform
`RouteOnAttribute`	Sends data to different paths based on conditions	Transform
`ListenHTTP`	Receives data sent over HTTP (from websites, IoT, etc.)	Source
`InvokeHTTP`	Calls websites or APIs to get or send data	Utility

🎉 Build Your First Flow: Generate → Log!

Loome kõige lihtsama voo, et näha NiFit töös!

Drag a Processor from the toolbar onto the canvas. Search for GenerateFlowFile and add it.
Drag another Processor. Search for LogAttribute and add it.
Connect them: Hover over GenerateFlowFile until you see a connection arrow. Drag it to LogAttribute. In the popup, check "success" and click "Create".
Configure GenerateFlowFile: Double-click it → go to "Scheduling" tab → set "Run Schedule" to 5 sec (nii genereerib see andmeid iga 5 sekundi järel).
Auto-terminate LogAttribute: Double-click LogAttribute → "Relationships" tab → check the box for "success" to auto-terminate.
Start both processors: Right-click each → "Start" (or select all and press the ▶ play button in the toolbar).

GenerateFlowFile
Creates test data

➜

📋 Järjekord

➜

LogAttribute
Logib konsooli

🎊 Congratulations!

You've just built your first NiFi data flow! Check the NiFi logs to see the FlowFile attributes being printed. You can find logs at: logs/nifi-app.log

📁 Second Flow: GetFile → PutFile (File Mover!)

Loome voo, mis automaatselt moves files from one folder to another!

Create two folders: ~/nifi-input ja ~/nifi-output
Lisa GetFile processor → configure "Input Directory" to /home/yourname/nifi-input
Lisa PutFile processor → configure "Directory" to /home/yourname/nifi-output
Connect GetFile → PutFile (relationship: "success")
Auto-terminate PutFile's "success" and "failure" relationships
Start both processors!

Terminal — Create test folders
# Loo sisend- ja väljundkaustadmkdir ~/nifi-input ~/nifi-output

# Loo testfailecho"Hello from Alin Li NiFi Course!" > ~/nifi-input/test.txt

# Jälgi väljundkaustals ~/nifi-output

Within seconds, the test file will magically appear in the output folder! 🪄

📋 Understanding Connections & Queues

The lines between processors are called Connections. Igal ühendusel on queue where FlowFiles wait.

💡 Think of it like a Conveyor Belt!

Imagine a conveyor belt between two workers. If Worker B is slower than Worker A, packages pile up on the belt. In NiFi, FlowFiles pile up in the queue. You can see exactly how many are waiting by looking at the number on the connection!

Queue features you should know:

Back Pressure: If a queue gets too full, NiFi automatically slows down the source processor. Smart!
Prioritisation: You can set rules like "oldest first" or "newest first".
Expiration: FlowFiles can be set to expire if they wait too long.

🎮 Activity: Match the Processor to its Job!

Drag each processor name to the correct description:

GetFile

PutFile

ListenHTTP

ReplaceText

Reads files from a folder

Writes data to a file

Receives HTTP requests

Finds & replaces text

🧪 Mooduli 3 viktoriin — vooehitaja!

1. Milline on lihtsa andmevoo õige järjestus?

A) Allikas → Ühendus → Teisendus → Ühendus → Sihtkoht

B) Destination → Source → Connection

C) Transform → Source → Destination

2. Mida teeb protsessor GenerateFlowFile?

A) Loeb faile kaustast

B) Sends data to a database

C) Creates test data for experimenting

3. Mis juhtub, kui NiFi järjekord täitub?

A) NiFi jookseb kokku

B) Back pressure slows down the source processor

C) All data is deleted

Moodul 4

🚀 Pärismaailma projektid

Apply your NiFi skills to exciting real projects — IoT sensors, web APIs, and file routing!

🌡️ Project 1: IoT Temperature Logger with ESP32

Collect temperature data from an ESP32 sensor and save it with NiFi!

The Idea: Your ESP32 reads a temperature sensor and sends the reading to NiFi over WiFi. NiFi receives it, adds a timestamp, and saves it to a file. This is exactly how real IoT systems work!

📟 ESP32 pool (Arduino IDE)

This sketch connects your ESP32 to WiFi and sends temperature to NiFi every 10 seconds:

Arduino IDE — ESP32 Sketch
// ============================================// Alin Li NiFi IoT Temperature Logger v1.0// ESP32 → NiFi HTTP POST-i kaudu// ============================================#include <WiFi.h>
#include <HTTPClient.h>

// WiFi sisselogimisandmedconst char* ssid = "AlinLi_NiFi";
const char* password = "12345678";

// NiFi ListenHTTP aadress// Asenda see IP oma NiFi serveri IP-aadressigaconst char* nifiURL = "http://192.168.4.1:9090/iot";

// Simuleeritud temperatuur (asenda päris anduriga)float readTemperature() {
  return20.0 + (random(0, 150) / 10.0);
}

voidsetup() { Serial.begin(115200);
  
  // Ühendu WiFi-ga
  WiFi.begin(ssid, password);
  Serial.print("Connecting to WiFi");
  while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println("\nConnected! IP: " + WiFi.localIP().toString()); }

voidloop() {
  if (WiFi.status() == WL_CONNECTED) { HTTPClient http; http.begin(nifiURL); http.addHeader("Content-Type", "application/json");
    
    // Loo JSON-andmesisufloat temp = readTemperature();
    String payload = "{\"device\":\"ESP32-01\",";
    payload += "\"temperature\":" + String(temp, 1) + ",";
    payload += "\"unit\":\"C\"}";
    
    // Saada NiFi-le POST-päringint httpCode = http.POST(payload);
    
    if (httpCode > 0) { Serial.println("Sent: " + payload + " → Response: " + String(httpCode)); } else {
      Serial.println("Error sending data!"); } http.end(); }
  
  // Saada iga 10 sekundi järel
  delay(10000);
}
      

🔄 NiFi pool — ehita see voog:

ListenHTTP
Port 9090, tee /iot

➜

UpdateAttribute
Lisa ajatempel

➜

PutFile
Save to folder

Lisa ListenHTTP → set Port to 9090 and Base Path to /iot
Lisa UpdateAttribute → add property: filename = temp_${now():format('yyyyMMdd_HHmmss')}.json
Lisa PutFile → set Directory to /home/yourname/iot-data
Connect them: ListenHTTP → UpdateAttribute → PutFile
Start all processors and run the ESP32!

📂 Tulemus

Every 10 seconds, a new JSON file will appear in your iot-data folder with the temperature reading from the ESP32! You're now running a real IoT data pipeline!

🌤️ Project 2: Weather Data Collector from API

Fetch weather data from a free API and save it automatically!

The Idea: NiFi küsib tasuta ilma-API-lt iga 5 minuti järel praegusi ilmaandmeid ja salvestab faili. Nii kogud aja jooksul ilmaajaloo andmebaasi!

🔄 NiFi voog:

GenerateFlowFile
Triggers every 5 min

➜

InvokeHTTP
Call weather API

➜

UpdateAttribute
Set filename

➜

PutFile
Save weather data

Lisa GenerateFlowFile → Scheduling: Run every 5 min
Lisa InvokeHTTP → set "HTTP URL" to a free weather API like:
https://api.open-meteo.com/v1/forecast?latitude=25.29&longitude;=51.53¤t;_weather=true
(See on Doha jaoks! Muuda koordinaadid oma linna omadeks)
Lisa UpdateAttribute → set filename: weather_${now():format('yyyyMMdd_HHmm')}.json
Lisa PutFile → Directory: /home/yourname/weather-data
Connect and start!

🆓 Open-Meteo API

Open-Meteo on täiesti tasuta avatud lähtekoodiga ilma-API — API-võtit pole vaja! See sobib suurepäraselt õppimiseks ja on kooskõlas meie avatud lähtekoodi väärtustega.

📂 Projekt 3: nutikas failisuunaja

Automatically sort files into folders based on their type!

The Idea: Drop any file into an "inbox" folder, and NiFi will automatically sort it — PDFs go to one folder, images to another, text files to a third!

🔄 NiFi voog:

GetFile
Watch inbox

➜

RouteOnAttribute
Check file type

➜

PutFile ×3
Sort into folders

Create folders: inbox, sorted/pdf, sorted/images, sorted/text
Lisa GetFile → Input Directory: your inbox folder
Lisa RouteOnAttribute → Add routing rules:
• pdf: ${filename:endsWith('.pdf')}
• images: ${filename:endsWith('.jpg'):or(${filename:endsWith('.png')})}
• text: ${filename:endsWith('.txt')}
Lisa 3 PutFile processors — one for each sorted folder
Connect RouteOnAttribute's "pdf" → PutFile (pdf folder), "images" → PutFile (images), "text" → PutFile (text)
Start and test by dropping different files into the inbox!

Terminal — Create test folders & files
# Loo kaustastruktuurmkdir-p ~/inbox ~/sorted/pdf ~/sorted/images ~/sorted/text

# Loo testfailidecho"Hello" > ~/inbox/readme.txt
touch ~/inbox/report.pdf
touch ~/inbox/photo.jpg

# Vaata, kuidas need automaatselt sorteeritakse!ls ~/sorted/pdf ~/sorted/images ~/sorted/text
      

🧠 NiFi avaldiste keel

The ${filename:endsWith('.pdf')} syntax is called NiFi Expression Language (EL). See on võimas minikeel FlowFile’i atribuutidega töötamiseks. Keerukamaid vooge ehitades kasutad seda sageli!

🌈 What's Next? Keep Exploring!

You've learned the fundamentals of Apache NiFi! Here are some exciting things to try next:

🗄️

Connect to Databases

Use PutDatabaseRecord to store data in MySQL, PostgreSQL, or SQLite

📧

Send Alerts

Use PutEmail to send yourself alerts when something interesting happens

🔗

Chain Multiple APIs

Get data from one API, transform it, and send it to another

📊

Build Dashboards

Connect NiFi to Grafana to visualise your IoT data in real-time

🧪 Lõpuviktoriin — projektimeister!

1. Milline protsessor võtab ESP32-lt vastu HTTP-päringuid?

A) InvokeHTTP

B) ListenHTTP

C) GetHTTP

2. Milline protsessor saadab API-le HTTP-päringuid?

A) ListenHTTP

B) GetFile

C) InvokeHTTP

3. Mida teeb RouteOnAttribute?

A) Saadab andmed tingimuste järgi eri radadele

B) Changes the content of a FlowFile

C) Deletes FlowFiles that match a pattern

4. Mida teeb NiFi avaldisekeeles ${filename:endsWith('.pdf')} check?

A) Kui faili sisu sisaldab sõna "pdf"

B) If the filename ends with ".pdf"

C) If the file size is large enough for a PDF

🏆 Course Complete!

Congratulations on completing the Alin Li NiFi kursus — andmevoogude seiklused!

You've learned how to install NiFi, build data flows, and create real IoT and API projects. Keep exploring, keep building, and remember:

„Avatud andmetöötluse vaimu taaselustamine“ 🌟