{ "id": "y0Yk7da21T4u9zlp", "meta": { "instanceId": "99f4e9e67f2a926c174453b6675a71cc5fb71c1fb19cfc06d50531053c661324", "templateCredsSetupCompleted": true }, "name": "Job Posting Aggregator with Email and GitHub", "tags": [], "nodes": [ { "id": "d55016ff-5fc3-4e1e-a4df-6a6f14f91997", "name": "Workflow Overview", "type": "n8n-nodes-base.stickyNote", "position": [ -272, 32 ], "parameters": { "width": 432, "height": 768, "content": "## How it works\n\nThis workflow lets professionals keep an eye on evolving certification or licence requirements without manually visiting every industry-body website. Start the workflow with the Manual Trigger and a Code node immediately produces a list of certification-issuer URLs. Each URL moves through a batch splitter so every page is handled one-by-one. ScrapeGraphAI then analyses the live web page and returns a clean JSON object that contains the certification name, renewal criteria, CE credit totals, cost and the current effective-date. The data is normalised in a Code node and compared with the last saved snapshot that lives in a GitHub repository. If a difference is detected—say the number of CE credits has changed—the new JSON snapshot overwrites the old file on GitHub and a concise email summary lands in your inbox. No change means no email, keeping noise to a minimum.\n\n## Setup steps\n\n1. Add your ScrapeGraphAI API credential in **Credentials / ScrapeGraphAI**\n2. Fill the URL list in the **Define Certification URLs** Code node\n3. Add a GitHub personal-access token and set owner/repo names in both GitHub nodes\n4. Configure **from** and **to** addresses in the Email Send node\n5. Commit and enable the workflow, then run the Manual Trigger to test\n6. (Optional) Replace the Manual Trigger with a Schedule Trigger for production use" }, "typeVersion": 1 }, { "id": "eb2185d9-7b55-44c6-81ab-3f1d7546e1a6", "name": "Section – Input & Batching", "type": "n8n-nodes-base.stickyNote", "position": [ 256, 16 ], "parameters": { "color": 7, "width": 514, "height": 718, "content": "## Input & Batching\n\nThe first three functional nodes work together to generate and serialise the list of certification URLs that will be monitored. The **Manual Trigger** gives you instant control so you can test the workflow on demand or wire it into a Schedule Trigger later for hands-free operation. Immediately after the trigger fires, the **Define Certification URLs** Code node returns an array of JSON items—each item holds a unique `url` and a human-readable `name`. These individual items travel to the **Split In Batches** node. Setting the batch-size to 1 forces n8n to process each certification page sequentially, preventing accidental rate-limit hits on ScrapeGraphAI and making downstream debugging easier because you see one certification at a time. If you add more URLs later, n8n will automatically loop through them without additional configuration, ensuring scalability while maintaining a straightforward linear flow." }, "typeVersion": 1 }, { "id": "1b93a9e7-c867-43cf-96cb-6076489b2fc9", "name": "Section – Scraping & Normalising", "type": "n8n-nodes-base.stickyNote", "position": [ 768, 32 ], "parameters": { "color": 7, "width": 546, "height": 670, "content": "## Scraping & Normalising\n\nOnce a single URL enters the Scrape stage the **ScrapeGraphAI** node takes centre stage. It loads the live web page of the certification body, executes any required JavaScript, and then—guided by a natural-language prompt—extracts structured details like renewal cycles, CE credit counts and associated fees. The AI returns data in a predictable JSON payload that feeds directly into the **Format Scraped Data** Code node. Here the workflow cleans null values, converts numbers, and generates a slugified filename that later becomes the unique key in GitHub (`data/slug.json`). The node also stamps the record with a `scraped_at` ISO timestamp for auditing. Normalising the data right away ensures downstream steps can rely on consistent field names and types, simplifying change-detection logic and keeping historical snapshots tidy inside the repository." }, "typeVersion": 1 }, { "id": "de0c3985-b53b-4477-a713-2763fcf9f644", "name": "Section – Change Detection", "type": "n8n-nodes-base.stickyNote", "position": [ 1408, 0 ], "parameters": { "color": 7, "width": 514, "height": 670, "content": "## Change Detection\n\nDetecting meaningful differences is the heart of the workflow. First, the **GitHub – Get Previous Snapshot** node fetches the prior JSON file that matches the slugified certification name. The content arrives Base64 encoded so the retrieval is lightweight. Both the fresh ScrapeGraphAI payload and the historic GitHub snapshot converge in a **Merge** node using the *combine* mode, effectively stitching both datasets into a single item. Inside **Detect Changes (Code)** the workflow decodes the stored Base64, converts it into a JavaScript object and performs a deep string comparison against the freshly scraped fields. The node then sets a simple boolean `hasChanges`. This flag drives the subsequent **IF** node that cleanly routes only modified certifications to the GitHub Update and email-alert path, thereby eliminating noise and keeping commit history meaningful." }, "typeVersion": 1 }, { "id": "9096f0da-830a-48b2-b4be-51549c7576dc", "name": "Section – Store & Notify", "type": "n8n-nodes-base.stickyNote", "position": [ 1984, 0 ], "parameters": { "color": 7, "width": 658, "height": 654, "content": "## Store & Notify\n\nWhen a change is confirmed, two final responsibilities kick in: persisting the new truth and alerting stakeholders. The **GitHub – Upsert Snapshot** node writes the freshly normalised JSON back to the repo, replacing or creating the corresponding file under `data/`. A concise commit message includes the certification name and timestamp, making it easy to audit the version history from GitHub’s web interface. Immediately after the commit, **Prepare Summary (Code)** crafts a plain-text diff highlighting what has changed—ideal for quick human scanning. Finally, **Email Send** delivers this summary straight to your inbox. Using email keeps the notification method universally accessible while GitHub provides a durable, version-controlled storage layer. If the IF node determines no updates occurred, execution simply ends, which means zero unnecessary commits and a silent inbox." }, "typeVersion": 1 }, { "id": "2f37bd1d-287c-407c-a6c4-2c29f6047002", "name": "Start Manual", "type": "n8n-nodes-base.manualTrigger", "position": [ 304, 400 ], "parameters": {}, "typeVersion": 1 }, { "id": "cb830a90-405a-4501-91d7-d57c29c4b74c", "name": "Define Certification URLs", "type": "n8n-nodes-base.code", "position": [ 512, 400 ], "parameters": { "jsCode": "// Add or modify certification pages here\nreturn [\n { json: { url: 'https://example.com/certification-a', name: 'Certification A' } },\n { json: { url: 'https://example.com/certification-b', name: 'Certification B' } }\n];" }, "typeVersion": 2 }, { "id": "47f9829a-dad9-48e0-bf5d-119be1612fc8", "name": "Split In Batches", "type": "n8n-nodes-base.splitInBatches", "position": [ 640, 416 ], "parameters": { "options": {} }, "typeVersion": 3 }, { "id": "4c137fa8-ce7e-451e-9188-453e78fea210", "name": "Scrape Certification Page", "type": "n8n-nodes-scrapegraphai.scrapegraphAi", "position": [ 912, 400 ], "parameters": { "userPrompt": "You are extracting certification renewal details for professionals. Return JSON: {\"certification\":\"string\",\"requirements\":\"string\",\"credits\":\"number\",\"cost\":\"string\",\"effective_date\":\"YYYY-MM-DD\"}. Pull only publicly available data.", "websiteUrl": "={{ $json.url }}" }, "typeVersion": 1 }, { "id": "ea18b7d6-129a-4930-b59d-f402b4404c61", "name": "Format Scraped Data", "type": "n8n-nodes-base.code", "position": [ 1104, 400 ], "parameters": { "jsCode": "const item = items[0].json;\n// Ensure required fields exist and build a slug for file naming\nconst slugSource = item.certification || item.name || 'unknown';\nconst slug = slugSource.toLowerCase().replace(/[^a-z0-9]+/g, '-').replace(/(^-|-$)/g, '');\nreturn [{\n json: {\n slug,\n certification: item.certification || item.name,\n requirements: item.requirements || '',\n credits: Number(item.credits || 0),\n cost: item.cost || 'N/A',\n effective_date: item.effective_date || null,\n scraped_at: new Date().toISOString()\n }\n}];" }, "typeVersion": 2 }, { "id": "6e4db302-8e4f-4e51-92b5-d300e9c27d9f", "name": "GitHub – Get Previous Snapshot", "type": "n8n-nodes-base.github", "position": [ 1312, 400 ], "webhookId": "4346bca3-472e-45fa-906a-d872af000c52", "parameters": { "owner": "YOUR_GITHUB_USERNAME", "filePath": "={{ 'data/' + $json.slug + '.json' }}", "resource": "file", "operation": "get", "repository": "certification-requirements-tracker", "additionalParameters": {} }, "typeVersion": 1 }, { "id": "510822e0-d738-43c4-b5fb-245c1fa8fd77", "name": "Merge Current & Previous", "type": "n8n-nodes-base.merge", "position": [ 1504, 400 ], "parameters": { "mode": "combine", "options": {}, "mergeByFields": { "values": [ {} ] } }, "typeVersion": 2 }, { "id": "9cbad78c-9385-43ff-a5c9-7434916d1747", "name": "Detect Changes", "type": "n8n-nodes-base.code", "position": [ 1712, 400 ], "parameters": { "jsCode": "const data = items[0].json;\nlet previousData = null;\nif (data.content) {\n try {\n previousData = JSON.parse(Buffer.from(data.content, 'base64').toString('utf8'));\n } catch (e) {\n previousData = null;\n }\n}\n// Build a comparable snapshot without volatile fields\nconst snapshot = {\n certification: data.certification,\n requirements: data.requirements,\n credits: data.credits,\n cost: data.cost,\n effective_date: data.effective_date\n};\nconst hasChanges = !previousData || JSON.stringify(previousData) !== JSON.stringify(snapshot);\nreturn [{\n json: {\n ...data,\n previousData,\n snapshot,\n hasChanges\n }\n}];" }, "typeVersion": 2 }, { "id": "a6981b53-d624-4c06-aff1-c963b84d5d74", "name": "Changes Detected?", "type": "n8n-nodes-base.if", "position": [ 1904, 400 ], "parameters": { "options": {}, "conditions": { "boolean": [ { "value1": "={{ $json.hasChanges }}", "value2": true } ] } }, "typeVersion": 2 }, { "id": "09f076b6-8ed0-43d6-8737-907f3c0f916b", "name": "GitHub – Upsert Snapshot", "type": "n8n-nodes-base.github", "position": [ 2112, 304 ], "webhookId": "ef8beaa4-ce97-4b1f-a391-0c5285f47b49", "parameters": { "owner": "YOUR_GITHUB_USERNAME", "filePath": "={{ 'data/' + $json.slug + '.json' }}", "resource": "file", "repository": "certification-requirements-tracker", "fileContent": "={{ Buffer.from(JSON.stringify($json.snapshot, null, 2)).toString('base64') }}", "commitMessage": "={{ 'Update requirements for ' + $json.certification + ' on ' + new Date().toISOString() }}" }, "typeVersion": 1 }, { "id": "15b7a912-4489-431a-b7c7-53a1a5ca9fca", "name": "Prepare Summary Email", "type": "n8n-nodes-base.code", "position": [ 2304, 304 ], "parameters": { "jsCode": "const d = items[0].json;\nlet body = `Certification: ${d.certification}\\n`;\nif (d.previousData) {\n body += `Previous Requirements: ${d.previousData.requirements}\\n`;\n body += `New Requirements: ${d.requirements}\\n`;\n body += `Credits: ${d.previousData.credits} ➜ ${d.credits}\\n`;\n body += `Cost: ${d.previousData.cost} ➜ ${d.cost}\\n`;\n body += `Effective Date: ${d.previousData.effective_date} ➜ ${d.effective_date}\\n`;\n} else {\n body += 'No previous record found. Initial snapshot created.';\n}\nreturn [{\n json: {\n emailSubject: `Certification Update – ${d.certification}`,\n emailBody: body\n }\n}];" }, "typeVersion": 2 }, { "id": "cb011d6b-1234-4bc6-b3e8-04514e354aaa", "name": "Email Send Notification", "type": "n8n-nodes-base.emailSend", "position": [ 2512, 304 ], "webhookId": "387f1e6d-8be3-4d16-b0c7-36f6cd8a9b1e", "parameters": { "options": {}, "subject": "={{ $json.emailSubject }}", "toEmail": "user@example.com", "fromEmail": "no-reply@example.com" }, "typeVersion": 2.1 } ], "active": false, "pinData": {}, "settings": { "executionOrder": "v1" }, "versionId": "5bf6a0f9-2298-420a-9251-ee8b136bfbcc", "connections": { "Start Manual": { "main": [ [ { "node": "Define Certification URLs", "type": "main", "index": 0 } ] ] }, "Detect Changes": { "main": [ [ { "node": "Changes Detected?", "type": "main", "index": 0 } ] ] }, "Split In Batches": { "main": [ [ { "node": "Scrape Certification Page", "type": "main", "index": 0 } ] ] }, "Changes Detected?": { "main": [ [ { "node": "GitHub – Upsert Snapshot", "type": "main", "index": 0 } ] ] }, "Format Scraped Data": { "main": [ [ { "node": "GitHub – Get Previous Snapshot", "type": "main", "index": 0 }, { "node": "Merge Current & Previous", "type": "main", "index": 0 } ] ] }, "Prepare Summary Email": { "main": [ [ { "node": "Email Send Notification", "type": "main", "index": 0 } ] ] }, "Merge Current & Previous": { "main": [ [ { "node": "Detect Changes", "type": "main", "index": 0 } ] ] }, "Define Certification URLs": { "main": [ [ { "node": "Split In Batches", "type": "main", "index": 0 } ] ] }, "Scrape Certification Page": { "main": [ [ { "node": "Format Scraped Data", "type": "main", "index": 0 } ] ] }, "GitHub – Upsert Snapshot": { "main": [ [ { "node": "Prepare Summary Email", "type": "main", "index": 0 } ] ] }, "GitHub – Get Previous Snapshot": { "main": [ [ { "node": "Merge Current & Previous", "type": "main", "index": 1 } ] ] } } }