افهم كيف تتدفق البيانات

الوحدة 1

🌟 What is Apache NiFi & Why It Matters?

Discover the amazing tool that moves data around like a super-smart delivery service!

📦 First — What is Data?

Data is information. إنها في كل مكان! اسمك ودرجاتك المدرسية ودرجة الحرارة في الخارج والصور على هاتفك والرسائل التي ترسلها — كل هذه تُعد data.

Companies and organisations collect tons of data every second. Think about a weather station recording temperature every minute, or a shop tracking what people buy. All this data needs to move from one place to another to be useful.

💡 Think of it like a Post Office!

Imagine millions of letters and packages arriving at a post office every day. Someone has to sort them, read the addresses, decide where they go, and deliver them. Apache NiFi does exactly this — but for data!

🐝 تعرّف على Apache NiFi!

Apache NiFi is a free, open-source software that helps you collect, move, and deliver data between different systems automatically. It was originally created by the NSA (وكالة الأمن القومي) في الولايات المتحدة، ثم قُدّم لاحقًا إلى مجتمع المصدر المفتوح عبر Apache Software Foundation.

The name "NiFi" comes from "NiagaraFiles" — like the powerful Niagara Falls, it handles huge rivers of data flowing from one place to another!

🔓 إنه مفتوح المصدر!

NiFi مجاني تمامًا للاستخدام والتعديل والمشاركة. ينسجم ذلك مع مهمة Alin Li لإحياء روح الحوسبة المفتوحة. لا تراخيص مدفوعة أو اشتراكات — فقط برنامج قوي مجاني للجميع!

🎯 What Can NiFi Do?

NiFi يشبه super-smart delivery service for data. إليك ما يمكنه فعله:

📥

Collect Data

Grab data from files, websites, sensors, databases, and more

🔄

Transform Data

Change data format, filter what you need, combine different sources

📤

Deliver Data

Send data to databases, files, cloud services, or other apps

👁️

راقب كل شيء

See exactly where your data is and what's happening to it in real-time

🔒

Keep It Safe

Encrypt data, control who can see it, and track every change

⚡

Handle Huge Loads

Process millions of data items per second without breaking a sweat

⚙️ How Does NiFi Work? (The Big Picture)

يستخدم NiFi مفهومًا يُسمّى «تدفقات البيانات». يشبه تدفّق البيانات وصفة تخبر NiFi: "خذ البيانات من هنا، ونفّذ عليها هذا، ثم أرسلها إلى هناك."

Here's a simple data flow:

📁 المصدر
Get the data

➜

⚙️ Process
Change it

➜

📋 طابور
Wait in line

➜

💾 الوجهة
Deliver it!

💡 Think of it like a Factory Assembly Line!

Raw materials (data) come in → workers on the line (processors) shape and transform them → finished products (processed data) come out at the end. NiFi is the factory manager that controls the entire line!

📖 Key Terms — Tap to Flip!

Learn the NiFi vocabulary by tapping each card:

📄

FlowFile
tap to flip

A FlowFile is a single piece of data moving through NiFi. It has content (the actual data) and attributes (labels describing it).

⚙️

Processor
tap to flip

A Processor is a worker that does something with data — reads a file, changes format, sends to a database, etc. NiFi has 300+ processors!

🔗

Connection
tap to flip

A Connection is the pipe between two processors. It holds a queue where FlowFiles wait before being processed.

🖥️

Canvas
tap to flip

The Canvas is the main workspace where you drag-and-drop processors and connections to build your data flow visually!

📊

Provenance
tap to flip

Provenance is NiFi's history book — it tracks every single thing that happened to every FlowFile. Perfect for debugging!

📂

Process Group
tap to flip

A Process Group is like a folder — it organises related processors together to keep your canvas clean and tidy.

🤔 Why Should YOU Care About NiFi?

Even as a young learner, understanding data flow is super valuable because:

IoT Projects: If you have an ESP32 or Arduino collecting sensor data, NiFi can receive and store it automatically!
Future-proof skills: Data engineering is one of the most in-demand jobs in the world right now.
Visual programming: يستخدم NiFi السحب والإفلات — لا حاجة لبرمجة معقّدة للبدء!
It's free and open source: مثالي للتعلّم دون إنفاق ريال واحد!

🧪 اختبار الوحدة 1 — اختبر معلوماتك!

1. إلى ماذا يرمز NiFi؟

A) ملفات واجهة الشبكة

B) NiagaraFiles

C) New Internet Flow Integration

2. ما FlowFile في NiFi؟

A) نوع من المعالجات

B) The main NiFi application

C) A single piece of data moving through NiFi

3. NiFi برنامج مفتوح المصدر. ماذا يعني ذلك؟

A) حر الاستخدام والتعديل والمشاركة

B) Only free for students

C) Requires a paid license

4. ما مساحة العمل Canvas في NiFi؟

A) قاعدة بيانات تُخزَّن فيها البيانات

B) The visual workspace where you build data flows

C) A type of connection

الوحدة 2

🛠️ التثبيت والإعداد

لنشغّل NiFi على حاسوبك — الأمر أسهل مما تظن!

📋 What You Need Before Starting

Before installing NiFi, make sure you have:

حاسوب running Linux (recommended!), Windows, or macOS.
Java 21 or newer installed. NiFi runs on Java, so this is essential.
At least 4 GB of RAM (8 GB أفضل للأداء السلس).
متصفح ويب — Firefox (our open-source favourite!) or Chromium.

🐧 Why Linux?

At Alin Li, we love Linux because it's free, open-source, and perfectly suited for running server applications like NiFi. Most NiFi installations in the real world run on Linux!

☕ Step 1: Install Java (OpenJDK)

يحتاج NiFi إلى Java ليعمل. سنثبّت OpenJDK — the free, open-source version of Java.

على Ubuntu/Debian Linux:

Terminal
# حدّث حزم نظامكsudoaptupdate# ثبّت OpenJDK 21sudoaptinstall openjdk-21-jdk

# تحقّق من التثبيتjava-version

You should see something like: openjdk version "21.0.x" — that means Java is ready!

🪟 تستخدم Windows؟

Download OpenJDK from adoptium.net — it's the free Java from the Eclipse Foundation. Run the installer and make sure to check "Set JAVA_HOME variable" during installation.

📥 Step 2: Download Apache NiFi

لننزّل NiFi نفسه الآن!

Go to the official site: nifi.apache.org
Click on «التنزيلات» in the navigation bar.
Download the Binary file (the .zip file for all platforms).
Extract the downloaded file to a folder you'll remember.

على Linux (باستخدام الطرفية):

Terminal
# انتقل إلى مجلدك الشخصيcd ~

# نزّل NiFi (تحقّق من أحدث إصدار على nifi.apache.org)wgethttps://downloads.apache.org/nifi/2.1.0/nifi-2.1.0-bin.zip# فك ضغط الأرشيفunzip nifi-2.1.0-bin.zip

# انتقل إلى مجلد NiFicd nifi-2.1.0

⚠️ Version Numbers Change!

The version number (2.1.0) will change over time. Always check the official NiFi website for the latest version before downloading.

🚀 Step 3: Start NiFi!

Time to launch NiFi and see the magic happen!

على Linux/macOS:

Terminal
# شغّل NiFi في الخلفية./bin/nifi.shstart# تحقّق من الحالة./bin/nifi.shstatus# لإيقاف NiFi لاحقًا:./bin/nifi.shstop

على Windows:

Command Prompt
REM Navigate to the NiFi bin foldercd C:\nifi-2.1.0\bin

REM Start NiFirun-nifi.bat

⏳ Be Patient!

يستغرق بدء NiFi نحو 30-60 ثانية. يحمّل في الخلفية معالجاته وأدواته القوية. انتظر ظهور "NiFi has started" في السجل.

🌐 Step 4: Open the NiFi Web Interface

لدى NiFi واجهة ويب جميلة. افتح المتصفح وانتقل إلى:

https://localhost:8443/nifi

يولّد NiFi 2.x اسم مستخدم وكلمة مرور عشوائيين عند أول تشغيل. تجدهما في السجل:

Terminal
# اعثر على بيانات الاعتماد المولّدةgrep-i"Generated Username" logs/nifi-app.log
grep-i"Generated Password" logs/nifi-app.log
    

🔒 HTTPS افتراضيًا

يستخدم NiFi 2.x اتصال HTTPS آمنًا افتراضيًا على المنفذ 8443. قد يعرض المتصفح تحذيرًا أمنيًا — هذا طبيعي لشهادة محلية موقّعة ذاتيًا. انقر «متقدم» ← «متابعة».

بعد تسجيل الدخول، سترى NiFi Canvas — your blank workspace ready for building data flows!

🗺️ Quick Tour of the NiFi Interface

The NiFi interface has these main parts:

📐

Toolbar

Top bar with icons for adding processors, labels, and other components

🎨

Canvas

The big white area where you drag-and-drop to build flows

📊

Status Bar

Shows how many processors are running, stopped, or have errors

⚙️

لوحة التشغيل

Controls to start, stop, and configure your data flow

🧪 اختبار الوحدة 2 — التحقّق من التثبيت!

1. ما الذي يجب تثبيته قبل NiFi؟

A) Python

B) Node.js

C) Java (OpenJDK)

2. ما عنوان الويب الافتراضي للوصول إلى NiFi؟

A) http://localhost:8080

B) https://localhost:8443/nifi

C) http://nifi.local:3000

3. أي أمر يبدأ NiFi على Linux؟

A) ./bin/nifi.sh start

B) nifi --run

C) sudo start nifi

الوحدة 3

🔗 بناء تدفّقات البيانات

Time to build your first data flow — drag, drop, connect, and watch data move!

🧱 The Building Blocks of a Data Flow

Every NiFi data flow is made from just a few building blocks:

📥 المصدر
Processor

➜

📋 الاتصال
(قائمة الانتظار)

➜

🔄 تحويل
Processor

➜

📋 الاتصال
(قائمة الانتظار)

➜

📤 الوجهة
Processor

That's it! With just processors and connections, you can build incredibly powerful data flows. Let's learn about each piece.

⚙️ Understanding Processors

Processors are the workers of NiFi. Each processor does ONE specific job. NiFi comes with over 300 معالج مدمج! إليك أهمها للمبتدئين:

Processor	What It Does	Category
`GetFile`	Reads files from a folder on your computer	Source
`PutFile`	Writes data to a file on your computer	Destination
`GenerateFlowFile`	Creates test data for experimenting	Source
`LogAttribute`	Prints FlowFile details to the log (for debugging)	Utility
`UpdateAttribute`	Changes or adds labels (attributes) to a FlowFile	Transform
`ReplaceText`	Finds and replaces text inside FlowFile content	Transform
`RouteOnAttribute`	Sends data to different paths based on conditions	Transform
`ListenHTTP`	Receives data sent over HTTP (from websites, IoT, etc.)	Source
`InvokeHTTP`	Calls websites or APIs to get or send data	Utility

🎉 Build Your First Flow: Generate → Log!

لننشئ أبسط تدفق ممكن لنرى NiFi أثناء العمل!

Drag a Processor from the toolbar onto the canvas. Search for GenerateFlowFile and add it.
Drag another Processor. Search for LogAttribute and add it.
Connect them: Hover over GenerateFlowFile until you see a connection arrow. Drag it to LogAttribute. In the popup, check "success" and click "Create".
Configure GenerateFlowFile: Double-click it → go to "Scheduling" tab → set "Run Schedule" to 5 sec (ليولّد بيانات كل 5 ثوانٍ).
Auto-terminate LogAttribute: Double-click LogAttribute → "Relationships" tab → check the box for "success" to auto-terminate.
Start both processors: Right-click each → "Start" (or select all and press the ▶ play button in the toolbar).

GenerateFlowFile
Creates test data

➜

📋 طابور

➜

LogAttribute
يسجّل في وحدة التحكم

🎊 Congratulations!

You've just built your first NiFi data flow! Check the NiFi logs to see the FlowFile attributes being printed. You can find logs at: logs/nifi-app.log

📁 Second Flow: GetFile → PutFile (File Mover!)

لننشئ تدفقًا ينفّذ تلقائيًا moves files from one folder to another!

Create two folders: ~/nifi-input و ~/nifi-output
إضافة GetFile processor → configure "Input Directory" to /home/yourname/nifi-input
إضافة PutFile processor → configure "Directory" to /home/yourname/nifi-output
Connect GetFile → PutFile (relationship: "success")
Auto-terminate PutFile's "success" and "failure" relationships
Start both processors!

Terminal — Create test folders
# أنشئ مجلدي الإدخال والإخراجmkdir ~/nifi-input ~/nifi-output

# أنشئ ملف اختبارecho"Hello from Alin Li NiFi Course!" > ~/nifi-input/test.txt

# راقب مجلد الإخراجls ~/nifi-output

Within seconds, the test file will magically appear in the output folder! 🪄

📋 Understanding Connections & Queues

The lines between processors are called Connections. لكل اتصال queue where FlowFiles wait.

💡 Think of it like a Conveyor Belt!

Imagine a conveyor belt between two workers. If Worker B is slower than Worker A, packages pile up on the belt. In NiFi, FlowFiles pile up in the queue. You can see exactly how many are waiting by looking at the number on the connection!

Queue features you should know:

Back Pressure: If a queue gets too full, NiFi automatically slows down the source processor. Smart!
Prioritisation: You can set rules like "oldest first" or "newest first".
Expiration: FlowFiles can be set to expire if they wait too long.

🎮 Activity: Match the Processor to its Job!

Drag each processor name to the correct description:

GetFile

PutFile

ListenHTTP

ReplaceText

Reads files from a folder

Writes data to a file

Receives HTTP requests

Finds & replaces text

🧪 اختبار الوحدة 3 — باني التدفّقات!

1. ما الترتيب الصحيح لتدفق بيانات أساسي؟

A) المصدر → الاتصال → التحويل → الاتصال → الوجهة

B) Destination → Source → Connection

C) Transform → Source → Destination

2. ماذا يفعل المعالج GenerateFlowFile؟

A) يقرأ الملفات من مجلد

B) Sends data to a database

C) Creates test data for experimenting

3. ماذا يحدث عندما يمتلئ طابور NiFi؟

A) يتعطل NiFi

B) Back pressure slows down the source processor

C) All data is deleted

الوحدة 4

🚀 مشاريع من الواقع

Apply your NiFi skills to exciting real projects — IoT sensors, web APIs, and file routing!

🌡️ Project 1: IoT Temperature Logger with ESP32

Collect temperature data from an ESP32 sensor and save it with NiFi!

The Idea: Your ESP32 reads a temperature sensor and sends the reading to NiFi over WiFi. NiFi receives it, adds a timestamp, and saves it to a file. This is exactly how real IoT systems work!

📟 جانب ESP32 (Arduino IDE)

This sketch connects your ESP32 to WiFi and sends temperature to NiFi every 10 seconds:

Arduino IDE — ESP32 Sketch
// ============================================// Alin Li NiFi IoT Temperature Logger v1.0// ESP32 → NiFi عبر HTTP POST// ============================================#include <WiFi.h>
#include <HTTPClient.h>

// بيانات اعتماد WiFiconst char* ssid = "AlinLi_NiFi";
const char* password = "12345678";

// عنوان ListenHTTP في NiFi// استبدل عنوان IP هذا بعنوان خادم NiFiconst char* nifiURL = "http://192.168.4.1:9090/iot";

// حرارة محاكاة (استبدلها بمستشعر حقيقي)float readTemperature() {
  return20.0 + (random(0, 150) / 10.0);
}

voidsetup() { Serial.begin(115200);
  
  // الاتصال بـ WiFi
  WiFi.begin(ssid, password);
  Serial.print("Connecting to WiFi");
  while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println("\nConnected! IP: " + WiFi.localIP().toString()); }

voidloop() {
  if (WiFi.status() == WL_CONNECTED) { HTTPClient http; http.begin(nifiURL); http.addHeader("Content-Type", "application/json");
    
    // إنشاء حمولة JSONfloat temp = readTemperature();
    String payload = "{\"device\":\"ESP32-01\",";
    payload += "\"temperature\":" + String(temp, 1) + ",";
    payload += "\"unit\":\"C\"}";
    
    // أرسل طلب POST إلى NiFiint httpCode = http.POST(payload);
    
    if (httpCode > 0) { Serial.println("Sent: " + payload + " → Response: " + String(httpCode)); } else {
      Serial.println("Error sending data!"); } http.end(); }
  
  // أرسل كل 10 ثوانٍ
  delay(10000);
}
      

🔄 جانب NiFi — ابنِ هذا التدفّق:

ListenHTTP
المنفذ 9090، المسار /iot

➜

UpdateAttribute
إضافة طابع زمني

➜

PutFile
Save to folder

إضافة ListenHTTP → set Port to 9090 and Base Path to /iot
إضافة UpdateAttribute → add property: filename = temp_${now():format('yyyyMMdd_HHmmss')}.json
إضافة PutFile → set Directory to /home/yourname/iot-data
Connect them: ListenHTTP → UpdateAttribute → PutFile
Start all processors and run the ESP32!

📂 النتيجة

Every 10 seconds, a new JSON file will appear in your iot-data folder with the temperature reading from the ESP32! You're now running a real IoT data pipeline!

🌤️ Project 2: Weather Data Collector from API

Fetch weather data from a free API and save it automatically!

The Idea: يستدعي NiFi واجهة طقس مجانية كل 5 دقائق، ويجلب البيانات الحالية ويحفظها بملف. مع الوقت تبني قاعدة بيانات لسجل الطقس!

🔄 تدفّق NiFi:

GenerateFlowFile
Triggers every 5 min

➜

InvokeHTTP
Call weather API

➜

UpdateAttribute
Set filename

➜

PutFile
Save weather data

إضافة GenerateFlowFile → Scheduling: Run every 5 min
إضافة InvokeHTTP → set "HTTP URL" to a free weather API like:
https://api.open-meteo.com/v1/forecast?latitude=25.29&longitude;=51.53¤t;_weather=true
(هذا خاص بالدوحة! غيّر الإحداثيات لتناسب مدينتك)
إضافة UpdateAttribute → set filename: weather_${now():format('yyyyMMdd_HHmm')}.json
إضافة PutFile → Directory: /home/yourname/weather-data
Connect and start!

🆓 Open-Meteo API

Open-Meteo واجهة API للطقس مجانية تمامًا ومفتوحة المصدر — لا تحتاج إلى مفتاح API! إنها مثالية للتعلّم ومتوافقة مع قيمنا في المصادر المفتوحة.

📂 المشروع 3: موجّه ملفات ذكي

Automatically sort files into folders based on their type!

The Idea: Drop any file into an "inbox" folder, and NiFi will automatically sort it — PDFs go to one folder, images to another, text files to a third!

🔄 تدفّق NiFi:

GetFile
Watch inbox

➜

RouteOnAttribute
Check file type

➜

PutFile ×3
Sort into folders

Create folders: inbox, sorted/pdf, sorted/images, sorted/text
إضافة GetFile → Input Directory: your inbox folder
إضافة RouteOnAttribute → Add routing rules:
• pdf: ${filename:endsWith('.pdf')}
• images: ${filename:endsWith('.jpg'):or(${filename:endsWith('.png')})}
• text: ${filename:endsWith('.txt')}
إضافة 3 PutFile processors — one for each sorted folder
Connect RouteOnAttribute's "pdf" → PutFile (pdf folder), "images" → PutFile (images), "text" → PutFile (text)
Start and test by dropping different files into the inbox!

Terminal — Create test folders & files
# أنشئ بنية المجلداتmkdir-p ~/inbox ~/sorted/pdf ~/sorted/images ~/sorted/text

# أنشئ ملفات اختبارecho"Hello" > ~/inbox/readme.txt
touch ~/inbox/report.pdf
touch ~/inbox/photo.jpg

# شاهدها تُفرز تلقائيًا!ls ~/sorted/pdf ~/sorted/images ~/sorted/text
      

🧠 لغة تعبيرات NiFi

The ${filename:endsWith('.pdf')} syntax is called NiFi Expression Language (EL). إنها لغة مصغّرة قوية للتعامل مع سمات FlowFile. ستستخدمها كثيرًا عند بناء تدفّقات أكثر تعقيدًا!

🌈 What's Next? Keep Exploring!

You've learned the fundamentals of Apache NiFi! Here are some exciting things to try next:

🗄️

Connect to Databases

Use PutDatabaseRecord to store data in MySQL, PostgreSQL, or SQLite

📧

Send Alerts

Use PutEmail to send yourself alerts when something interesting happens

🔗

Chain Multiple APIs

Get data from one API, transform it, and send it to another

📊

Build Dashboards

Connect NiFi to Grafana to visualise your IoT data in real-time

🧪 الاختبار النهائي — محترف المشاريع!

1. أي معالج يستقبل طلبات HTTP من ESP32؟

A) InvokeHTTP

B) ListenHTTP

C) GetHTTP

2. أي معالج يرسل طلبات HTTP إلى API؟

A) ListenHTTP

B) GetFile

C) InvokeHTTP

3. ماذا يفعل RouteOnAttribute؟

A) يرسل البيانات إلى مسارات مختلفة حسب الشروط

B) Changes the content of a FlowFile

C) Deletes FlowFiles that match a pattern

4. في لغة تعبيرات NiFi، ما الذي يفعله ${filename:endsWith('.pdf')} check?

A) إذا كان محتوى الملف يتضمن "pdf"

B) If the filename ends with ".pdf"

C) If the file size is large enough for a PDF

🏆 Course Complete!

Congratulations on completing the Alin Li — دورة NiFi — مغامرات تدفق البيانات!

You've learned how to install NiFi, build data flows, and create real IoT and API projects. Keep exploring, keep building, and remember:

«إحياء روح الحوسبة المفتوحة» 🌟