Skip to content

ETL to QE, Update 66, Do One Thing and Do It Well

Date: 2025-04-05

Observing

Here is a narrative,

  • A mentor shows up to help you
  • The mentor asks you to make a list of the 10 most important things you want to accomplish
  • You present the list your Mentor
  • The mentor tell you to choose the three most important items from that list
  • You numerate the three most important items one through three on the list and present it back to your mentor
  • The mentor then tells you to cross out every thing but the most important item and ignore them until that is accomplished

I have a variety of lists but my DDaemon 2025 Roadmap Rev. 0.0.4 should fit the bill of the long list.

Here are the top 3 items, 1. Nostr Scraping Project 2. Nostr CMS 3. Nostr Bot Daemon Framework

Therefore the number one item is, 1. Nostr Scraping Project

Orienting

Now here's the trick of this exercise. If you are only allowed to focus on one thing it forces the user to decide weather that item actually makes sense to be the most important.

So in my case I can go scrape a couple million nostr events... But then what that's just the first step. That step does not resonate with the other parts of the Dentropy Daemon roadmap.

Well the goal of the Dentropy Daemon is to be a Sense Making Apparatus. Right now on Nostr there is no good media curation. People join the platform and leave because all they see is a mess of posts and bitcoin masturbation.

Therefore my goal is not to really Scrape Nostr but to develop a interface to group and label events on Nostr.

There are a variety of ways to label stuff on Nostr,

  • Use a specific account to interact with events
  • Use hashtags
  • Nostr Communities are a Thing NIP-72
  • Use lists
  • Define a new NIP and with specific tag and event kind

Perspective

But that is still Losing the forest for the trees, my Dentropy Daemon Tickets note (NOT Publicly Shared), has plastered on the top of it USER JOURNEY VALIDATION.

So the story here is this,

  • Scrape a tone of Nostr Events and put them on my own Nostr Relay
  • Use or Develop my own NIP to label, sort, and make sense of Nostr Event
  • Develop my own Nostr Client to make sense of the media

Well that is actually a list of three important things,

  1. Scrape a tone of Nostr Events and put them on my own Nostr Relay
  2. Use or Develop my own NIP to label, sort, and make sense of Nostr Event
  3. Develop my own Nostr Client to make sense of the media

So we just want to scrape a tone of Nostr Events. We just need to run our own relay and write a simple script. Once we have 1,000,000 events we can call the quest sucessful.

But what method are we going to use to Scape Events?

Argue with Myself to Decide

I am going to make a list of Account NPUB's I personally find varied and interesting.

Then develop a script that scrapes all their events, dumps then to a file, removes duplicates.

What about metadata of Profiles such as what relay has what data.

Or can we have a separate file for each relay?

Well we are going to have to iterate on this, and write a couple project updates.

We can then grab the follow list of each profile, store it in a sorted list JSON file, and scrape everything.

But how do we track what Relay's we have scraped or not.

We need a Nostr Scraping NIP.

Is that within Scope.

It has been on on the tip of my subconscious for literal months.

Decide

So what is the user journey here?

  • We start with a single NIP05
  • We get the JSON of the User and store it as an Event
    • Sha256 the content please
  • We then scrape every event from that NIP05 NPUB to a Database
  • We then grab all the
  • We then look at their follow list
  • We add every NPUB to a backlog of Nostr events to scrape

So what about tracking the metadata such as what relay has what event?

That kind of complexity and scope creep is what has paralyzed myself over the last couple months.

Wait we just need to produce Logs, every event that comes through a Nostr filter just needs to be logged problem solved.

Okay and for tooling what are we doing to use?

And what nostr NPUB are we going to bootstrap from,

Act

Sources